HUB
Sign inSign up

dustynv/transformers

By dustynv

•Updated almost 3 years ago

https://github.com/dusty-nv/jetson-containers/packages/llm/transformers

Image
0

4.9K

dustynv/transformers repository overview

⁠transformers

The HuggingFace Transformers⁠ library supports a wide variety of NLP and vision models with a convenient API, and is used by many of the other LLM packages. There are a large number of models that it's compatible with on HuggingFace Hub⁠.

Note

If you wish to use Transformer's integrated [bitsandbytes](https://huggingface.co/docs/transformers/main_classes/quantization#bitsandbytes-integration) quantization (`load_in_8bit/load_in_4bit`) or [AutoGPTQ](https://huggingface.co/docs/transformers/main_classes/quantization#autogptq-integration) quantization, run these containers instead which include those respective libraries installed on top of Transformers: * [`auto_gptq`](https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/auto_gptq) (depends on Transformers) * [`bitsandbytes`](https://github.com/dusty-nv/jetson-containers/tree/master/packages/llm/bitsandbytes) (depends on Transformers)
⁠Text Generation Benchmark

Substitute the text-generation model⁠ that you want to run (it should be a CausalLM model like GPT, Llama, ect)

./run.sh $(./autotag transformers) \
   huggingface-benchmark.py --model=gpt2

If the model repository is private or requires authentication, add --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN>

By default, the performance is measured for generating 128 new output tokens (this can be set with --tokens=N)

The prompt can be changed with --prompt='your prompt here'

⁠Precision / Quantization

Use the --precision argument to enable quantization (options are: fp32 fp16 fp4 int8, default is: fp16)

If you're using fp4 or int8, run the bitsandbytes⁠ container as noted above, so that bitsandbytes package is installed to do the quantization. It's expected that 4-bit/8-bit quantization is slower through Transformers than FP16 (while consuming less memory) - see here⁠ for more info.

Other libraries like exllama⁠, awq⁠, and AutoGPTQ⁠ have custom CUDA kernels and more efficient quantized performance.

⁠Llama2
./run.sh --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN> $(./autotag transformers) \
   huggingface-benchmark.py --model=meta-llama/Llama-2-7b-hf
CONTAINERS⁠
transformers
   Buildstransformers_jp60 transformers_jp51 transformers_jp46
   RequiresL4T ['>=32.6']
   Dependenciesbuild-essential⁠ cuda⁠ cudnn⁠ python⁠ numpy⁠ cmake⁠ onnx⁠ pytorch:2.2⁠ torchvision⁠ huggingface_hub⁠ rust⁠
   Dependantsaudiocraft⁠ auto_awq:0.2.4⁠ auto_gptq:0.7.1⁠ awq:0.1.0⁠ bitsandbytes⁠ bitsandbytes:builder⁠ efficientvit⁠ gptq-for-llama⁠ l4t-diffusion⁠ llava⁠ mlc:0.1.0⁠ mlc:0.1.0-builder⁠ mlc:0.1.1⁠ mlc:0.1.1-builder⁠ nanodb⁠ nanoowl⁠ nanosam⁠ nemo⁠ optimum⁠ stable-diffusion⁠ stable-diffusion-webui⁠ tensorrt_llm:0.10.dev0⁠ tensorrt_llm:0.10.dev0-builder⁠ tensorrt_llm:0.5⁠ tensorrt_llm:0.5-builder⁠ text-generation-inference⁠ text-generation-webui:1.7⁠ text-generation-webui:6a7cd01⁠ text-generation-webui:main⁠ voicecraft⁠ whisperx⁠ xtts⁠
   DockerfileDockerfile⁠
   Imagesdustynv/transformers:git-r35.2.1⁠ (2023-12-15, 5.9GB)
dustynv/transformers:git-r35.3.1⁠ (2023-12-12, 5.9GB)
dustynv/transformers:git-r35.4.1⁠ (2023-12-11, 5.9GB)
dustynv/transformers:nvgpt-r35.2.1⁠ (2023-12-05, 5.9GB)
dustynv/transformers:nvgpt-r35.3.1⁠ (2023-12-15, 5.9GB)
dustynv/transformers:nvgpt-r35.4.1⁠ (2023-12-14, 5.9GB)
dustynv/transformers:r32.7.1⁠ (2023-12-15, 1.5GB)
dustynv/transformers:r35.2.1⁠ (2023-12-11, 5.9GB)
dustynv/transformers:r35.3.1⁠ (2023-12-12, 5.9GB)
dustynv/transformers:r35.4.1⁠ (2023-12-15, 5.9GB)
dustynv/transformers:r36.2.0⁠ (2023-12-15, 7.6GB)
   Notesbitsandbytes and auto_gptq dependencies added on JetPack5 for 4-bit/8-bit quantization
transformers:git
   Buildstransformers-git_jp51
   RequiresL4T ['>=32.6']
   Dependenciesbuild-essential⁠ cuda⁠ cudnn⁠ python⁠ numpy⁠ cmake⁠ onnx⁠ pytorch:2.2⁠ torchvision⁠ huggingface_hub⁠ rust⁠
   DockerfileDockerfile⁠
   Imagesdustynv/transformers:git-r35.2.1⁠ (2023-12-15, 5.9GB)
dustynv/transformers:git-r35.3.1⁠ (2023-12-12, 5.9GB)
dustynv/transformers:git-r35.4.1⁠ (2023-12-11, 5.9GB)
   Notesbitsandbytes and auto_gptq dependencies added on JetPack5 for 4-bit/8-bit quantization
transformers:nvgpt
   Buildstransformers-nvgpt_jp51
   RequiresL4T ['>=32.6']
   Dependenciesbuild-essential⁠ cuda⁠ cudnn⁠ python⁠ numpy⁠ cmake⁠ onnx⁠ pytorch:2.2⁠ torchvision⁠ huggingface_hub⁠ rust⁠
   DockerfileDockerfile⁠
   Imagesdustynv/transformers:nvgpt-r35.2.1⁠ (2023-12-05, 5.9GB)
dustynv/transformers:nvgpt-r35.3.1⁠ (2023-12-15, 5.9GB)
dustynv/transformers:nvgpt-r35.4.1⁠ (2023-12-14, 5.9GB)
   Notesbitsandbytes and auto_gptq dependencies added on JetPack5 for 4-bit/8-bit quantization
CONTAINER IMAGES⁠
Repository/TagDateArchSize
  dustynv/transformers:git-r35.2.1⁠2023-12-15arm645.9GB
  dustynv/transformers:git-r35.3.1⁠2023-12-12arm645.9GB
  dustynv/transformers:git-r35.4.1⁠2023-12-11arm645.9GB
  dustynv/transformers:nvgpt-r35.2.1⁠2023-12-05arm645.9GB
  dustynv/transformers:nvgpt-r35.3.1⁠2023-12-15arm645.9GB
  dustynv/transformers:nvgpt-r35.4.1⁠2023-12-14arm645.9GB
  dustynv/transformers:r32.7.1⁠2023-12-15arm641.5GB
  dustynv/transformers:r35.2.1⁠2023-12-11arm645.9GB
  dustynv/transformers:r35.3.1⁠2023-12-12arm645.9GB
  dustynv/transformers:r35.4.1⁠2023-12-15arm645.9GB
  dustynv/transformers:r36.2.0⁠2023-12-15arm647.6GB

Container images are compatible with other minor versions of JetPack/L4T:
    • L4T R32.7 containers can run on other versions of L4T R32.7 (JetPack 4.6+)
    • L4T R35.x containers can run on other versions of L4T R35.x (JetPack 5.1+)

RUN CONTAINER⁠

To start the container, you can use jetson-containers run⁠ and autotag⁠, or manually put together a docker run⁠ command:

# automatically pull or build a compatible container image
jetson-containers run $(autotag transformers)

# or explicitly specify one of the container images above
jetson-containers run dustynv/transformers:nvgpt-r35.3.1

# or if using 'docker run' (specify image and mounts/ect)
sudo docker run --runtime nvidia -it --rm --network=host dustynv/transformers:nvgpt-r35.3.1

jetson-containers run⁠ forwards arguments to docker run⁠ with some defaults added (like --runtime nvidia, mounts a /data cache, and detects devices)
autotag⁠ finds a container image that's compatible with your version of JetPack/L4T - either locally, pulled from a registry, or by building it.

To mount your own directories into the container, use the -v⁠ or --volume⁠ flags:

jetson-containers run -v /path/on/host:/path/in/container $(autotag transformers)

To launch the container running a command, as opposed to an interactive shell:

jetson-containers run $(autotag transformers) my_app --abc xyz

You can pass any options to it that you would to docker run⁠, and it'll print out the full command that it constructs before executing it.

BUILD CONTAINER⁠
⁠

If you use ⁠autotag⁠ as shown above, it'll ask to build the container for you if needed. To manually build it, first do the system setup⁠, then run:

jetson-containers build transformers

The dependencies from above will be built into the container, and it'll be tested during. Run it with --help⁠ for build options.

Tag summary

Content type

Image

Digest

sha256:002dbc33d…

Size

5.9 GB

Last updated

almost 3 years ago

docker pull dustynv/transformers:nvgpt-r35.3.1