dustynv/local_llm repository overview

• Optimized LLM inference engine with support for AWQ and MLC quantization, multimodal agents, and live ASR/TTS.
• Web UI server using Flask, WebSockets, WebAudio, HTML5, Bootstrap5.
• Modes to run: Text Chat⁠, Multimodal Chat⁠, Voice Chat⁠, Live Llava⁠

⁠Text Chat

As an initial example, first test the console-based chat demo from chat/__main__.py⁠

./run.sh --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN> \
  $(./autotag local_llm) \
    python3 -m local_llm.chat --api=mlc \
      --model=meta-llama/Llama-2-7b-chat-hf

The model will automatically be quantized the first time it's loaded (in this case, with MLC and 4-bit). Other fine-tuned versions of Llama that have the same architecture (or are supported by the quantization API you have selected) should be compatible - see here⁠ for MLC.

⁠Command-Line Options

Some of the noteworthy command-line options can be found in utils/args.py⁠

⁠Automated Prompts

During testing, you can specify prompts on the command-line that will run sequentially:

./run.sh --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN> \
  $(./autotag local_llm) \
    python3 -m local_llm.chat --api=mlc \
      --model=meta-llama/Llama-2-7b-chat-hf \
      --prompt 'hi, how are you?' \
      --prompt 'whats the square root of 900?' \
      --prompt 'whats the previous answer times 4?' \
      --prompt 'can I get a recipie for french onion soup?'

You can also load JSON files containing prompt sequences, like from /data/prompts/qa.json⁠

⁠Multimodal Chat

If you load a Llava vision-language model, you can enter image files into the prompt, followed by questions about them:

./run.sh $(./autotag local_llm) \
  python3 -m local_llm.chat --api=mlc \
    --model=liuhaotian/llava-v1.5-13b \
    --prompt '/data/images/fruit.jpg' \
    --prompt 'what kind of fruits do you see?' \
    --prompt 'reset' \
    --prompt '/data/images/dogs.jpg' \
    --prompt 'what breed of dogs are in the image?' \
    --prompt 'reset' \
    --prompt '/data/images/path.jpg' \
    --prompt 'what does the sign say?'

Llava was trained to converse about one image at a time, hence the chat history is reset between images (otherwise the model tends to combine the features of all the images in the chat so far). Multiple questions can be asked about each image though.

By omitting --prompt, you can chat interactively from the terminal. If you enter an image filename, it will load that image, and then asking you for the prompt. Entering clear or reset will reset the chat history.

⁠Voice Chat

To enable the web UI and ASR/TTS for live conversations, follow the steps below.

⁠Start Riva Server

The ASR and TTS services use NVIDIA Riva with audio transformers and TensorRT. The Riva server runs locally in it's own container. Follow the steps from the riva-client:python⁠ package to run and test the Riva server on your Jetson.

1. Start the Riva server on your Jetson by following riva_quickstart_arm64⁠
2. Run some of the Riva ASR examples to confirm that ASR is working: https://github.com/nvidia-riva/python-clients#asr⁠
3. Run some of the Riva TTS examples to confirm that TTS is working: https://github.com/nvidia-riva/python-clients#tts⁠

You can also see this helpful video and guide from JetsonHacks for setting up Riva: Speech AI on Jetson Tutorial⁠

⁠Enabling HTTPS/SSL

Browsers require HTTPS in order to access the client's microphone. A self-signed SSL certificate was built into the container like this:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365 -nodes -subj '/CN=localhost'

The container's certificate is found under /etc/ssl/private and is automatically used, so HTTPS/SSL is enabled by default for these web UI's (you can change the PEM certificate/key used by setting the SSL_KEY and SSL_CERT environment variables). When you first navigate your browser to a page that uses these self-signed certificates, it will issue you a warning since they don't originate from a trusted authority:

You can choose to override this, and it won't re-appear again until you change certificates or your device's hostname/IP changes.

⁠Start Web Agent

./run.sh \
  --env HUGGINGFACE_TOKEN=<YOUR-ACCESS-TOKEN> \
  $(./autotag local_llm) \
    python3 -m local_llm.agents.web_chat --api=mlc \
      --model meta-llama/Llama-2-7b-chat-hf

You can then navigate your web browser to https://HOSTNAME:8050 and unmute your microphone.

• The default port is 8050, but can be changed with --web-port (and --ws-port for the websocket port)
• To debug issues with client/server communication, use --web-trace to print incoming/outgoing websocket messages.
• During bot replies, the TTS model will pause output if you speak a few words in the mic to interrupt it.
• If you loaded a multimodal Llava model instead, you can drag-and-drop images from the client.

⁠Live Llava

The VideoQuery⁠ agent processes an incoming camera or video feed on prompts in a closed loop with Llava. Navigate your browser to https://<IP_ADDRESS>:8050 after launching it, proceed past the SSL warning⁠, and see this demo walkthrough⁠ video on using the web UI.

./run.sh $(./autotag local_llm) \
  python3 -m local_llm.agents.video_query --api=mlc \
    --model Efficient-Large-Model/VILA-2.7b \
    --max-context-len 768 \
    --max-new-tokens 32 \
    --video-input /dev/video0 \
    --video-output webrtc://@:8554/output

This uses jetson_utils⁠ for video I/O, and for options related to protocols and file formats, see Camera Streaming and Multimedia⁠. In the example above, it captures a V4L2 USB webcam connected to the Jetson (under the device /dev/video0) and outputs a WebRTC stream.

⁠Processing a Video File or Stream

The example above was running on a live camera, but you can also read and write a video file or stream⁠ by substituting the path or URL to the --video-input and --video-output command-line arguments like this:

./run.sh \
  -v /path/to/your/videos:/mount
  $(./autotag local_llm) \
    python3 -m local_llm.agents.video_query --api=mlc \
      --model Efficient-Large-Model/VILA-2.7b \
      --max-context-len 768 \
      --max-new-tokens 32 \
      --video-input /mount/my_video.mp4 \
      --video-output /mount/output.mp4 \
      --prompt "What does the weather look like?"

This example processes and pre-recorded video (in MP4, MKV, AVI, FLV formats with H.264/H.265 encoding), but it also can input/output live network streams like RTP⁠, RTSP⁠, and WebRTC⁠ using Jetson's hardware-accelerated video codecs.

⁠NanoDB Integration

If you launch the VideoQuery⁠ agent with the --nanodb flag along with a path to your NanoDB database, it will perform reverse-image search on the incoming feed against the database by re-using the CLIP embeddings generated by the VLM.

To enable this mode, first follow the NanoDB tutorial⁠ to download, index, and test the database. Then launch VideoQuery like this:

./run.sh $(./autotag local_llm) \
  python3 -m local_llm.agents.video_query --api=mlc \
    --model Efficient-Large-Model/VILA-2.7b \
    --max-context-len 768 \
    --max-new-tokens 32 \
    --video-input /dev/video0 \
    --video-output webrtc://@:8554/output \
    --nanodb /data/nanodb/coco/2017

You can also tag incoming images and add them to the database using the panel in the web UI:

To start the container, you can use jetson-containers run⁠ and autotag⁠, or manually put together a docker run⁠ command:

# automatically pull or build a compatible container image
jetson-containers run $(autotag local_llm)

# or explicitly specify one of the container images above
jetson-containers run dustynv/local_llm:r35.4.1

# or if using 'docker run' (specify image and mounts/ect)
sudo docker run --runtime nvidia -it --rm --network=host dustynv/local_llm:r35.4.1

To mount your own directories into the container, use the -v⁠ or --volume⁠ flags:

jetson-containers run -v /path/on/host:/path/in/container $(autotag local_llm)

To launch the container running a command, as opposed to an interactive shell:

jetson-containers run $(autotag local_llm) my_app --abc xyz

You can pass any options to it that you would to docker run⁠, and it'll print out the full command that it constructs before executing it.

If you use ⁠autotag⁠ as shown above, it'll ask to build the container for you if needed. To manually build it, first do the system setup⁠, then run:

jetson-containers build local_llm

The dependencies from above will be built into the container, and it'll be tested during. Run it with --help⁠ for build options.