zahamm/llama-cpp-jetson repository overview

⁠llama-cpp-jetson (JetPack 6 / Orin Nano)

Prebuilt llama.cpp server optimized for NVIDIA Jetson Orin Nano running JetPack 6 (L4T r36.x) with CUDA 12.6.

This image provides a ready-to-run llama-server with GPU acceleration, tuned specifically for Jetson’s unified memory architecture.

⁠Supported Platforms

• NVIDIA Jetson Orin Nano (8GB or 16GB)
• JetPack 6.x (L4T r36.x)
• CUDA 12.6
• ARM64 only (not x86 compatible)

⁠Image Tags

⁠Requirements

• JetPack 6 installed
• NVIDIA Container Runtime (nvidia-container-runtime)
• Docker configured with NVIDIA support

Verify NVIDIA runtime:

docker info | grep -i nvidia

⁠Quick Start (docker-compose.yaml for Orin NX 16GB, use a different smaller model for Nano Super 8GB)

services:
  llama:
    image: zahamm/llama-cpp-jetson:latest
    container_name: llama_cpp
    runtime: nvidia
    network_mode: host
    restart: unless-stopped

    volumes:
      - ./models:/models:ro

    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility

    command: >
      --model /models/Qwen3.5-9B-Q5_K_M.gguf
      --mmproj /models/mmproj-F16.gguf
      --host 0.0.0.0
      --port 8080
      --reasoning on
      --spec-type draft-mtp
      --spec-draft-n-max 2
      --n-gpu-layers 99
      --ctx-size 32768
      --mmap
      --parallel 1
      --image-min-tokens 1024
      -fa on
      --cache-type-k q8_0
      --cache-type-v q8_0
      --temp 0.25
      --repeat_penalty 1.18
      --repeat_last_n 512
      --top_p 0.8
      --top_k 30

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]

Start the container:

docker compose up -d

Health check:

curl http://localhost:8080/health

Start using the WebUI:

http://your-jetson-host-name:8080

⁠Recommended Models (Hugging Face)

⁠Download and copy to your models/ directory wherever docker-compose.yml is running.

⁠Best Overall (Orin Nano Sweet Spot)

Llama-3.2-1B-Instruct

https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct⁠

Recommended quantization:

• Q5_K_M

Approximate RAM usage:

• 2.5–3 GB total

⁠Larger but Still Usable

unsloth/Qwen3-4B-GGUF

https://huggingface.co/unsloth/Qwen3-4B-GGUF⁠

Recommendations:

• Quantization: Q3_K_S
• GPU layers: ~22
• Context size: ≤ 2048

Approximate RAM usage:

• 3–4 GB

Llama-3.2-3B-Instruct

https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct⁠

Recommendations:

• Quantization: Q4_K_M
• GPU layers: ~20
• Context size: ≤ 2048

Approximate RAM usage:

• 4–5 GB

⁠Experimental / Advanced

Mistral-7B

Recommendations:

• Quantization: Q2_K or Q3_K
• GPU layers: ≤ 12
• Context size: ≤ 2048

Stability varies depending on workload and memory pressure. This will be REALLY slow.

⁠Memory & Performance Notes (Important)

Jetson Orin Nano uses unified system memory (shared between CPU and GPU).

Key guidelines:

• Do not offload all layers to GPU
• Keep context size ≤ 2048
• Always use --mmap
• Avoid running a desktop environment if headless (sudo systemctl set-default multi-user.target; sudo reboot)
• Don't run other services (apache, mysql, k8s, etc) that use memory.

Practical limits:

• Performance: Expect 4-9 tokens per second once model is loaded.
• 1B–4B models: stable and recommended
• 7B models: experimental
• Large context + large models: not supported

⁠API Compatibility

The server exposes a llama.cpp-compatible HTTP API:

• /completion
• /v1/chat/completions (OpenAI-compatible)
• /health

This allows drop-in use with many OpenAI-style clients and tools.

⁠License

• llama.cpp: MIT License
• Models: See individual Hugging Face licenses