n8500x/llamacpp-jetson repository overview

⁠llamacpp-jetson

The n8500x/llamacpp engine image built with CUDA for NVIDIA Jetson Orin (arm64, GPU sm_87, JetPack 6 / L4T r36). Same entrypoint, same OpenAI-compatible API, same env contract as the CPU image — plus the model runs on the GPU.

• Image: n8500x/llamacpp-jetson on Docker Hub (arm64 only). Tags: latest, YYYYMMDD-HHMMSS (the build stamp), and the llama.cpp release tag it was built from (b10884). Every tag on the Hub was built and run on a Jetson first.
• Pinned tag: b10884 (build arg LLAMA_TAG), base nvcr.io/nvidia/l4t-jetpack:r36.4.0 for the build stage only; the runtime stage is plain Ubuntu 22.04 + the CUDA runtime libraries. The GPU driver is injected by the NVIDIA container runtime.
• CPU code is pinned to the Orin's Cortex-A78AE (CPU_ARCH=armv8.2-a+fp16+dotprod), so the image is identical whether the Jetson or a GitHub runner built it.
• Needs: --runtime nvidia (or runtime: nvidia in compose). Without it the CUDA backend cannot find libcuda.so.1 and the server falls back to CPU.

⁠Serve

docker compose up -d                     # on the Jetson, models in ../models
curl -s http://localhost:8080/v1/models
curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-4b","messages":[{"role":"user","content":"hello"}],"max_tokens":64}'

Defaults differ from the CPU image where 8 GB of shared memory demands it: CTX_SIZE=8192, BATCH_SIZE=512, UBATCH_SIZE=256, GPU offload of all layers (LLAMA_ARG_N_GPU_LAYERS=99), flash attention on. The compose file also quantises the KV cache to q8_0, keeps the host prompt cache at 512 MiB and runs one slot. An 8B Q4_K_M model fits with those settings on an Orin Nano 8 GB; a 4B model leaves room for a larger context or two slots.

⁠Benchmark

docker run --rm --runtime nvidia -v ~/models:/models:ro n8500x/llamacpp-jetson \
  llama-bench -m /models/Qwen3-4B-Instruct-2507-Q4_K_M.gguf -ngl 99 -p 128 -n 32

⁠Build

GitHub Actions (the llamacpp-jetson GitHub Actions workflow in the docker repository) builds the image on GitHub's arm64 runners whenever this directory, entrypoint.sh or the workflow changes, and on Run workflow: a check that the Dockerfile still compiles and the binaries start. No GPU is needed to compile (nvcc comes from the JetPack image), nothing is pushed, and it takes one to two hours on the 2-vCPU runner.

On the Jetson is where the real image comes from (30 to 60 minutes; the build starves a running model server, so stop it first): from the laptop, ..\build-jetson.ps1 copies the build context over SSH and runs build.sh there (nohup, log in ~/llamacpp-jetson/build.log), tagging n8500x/llamacpp-jetson:latest, :<timestamp> and the old local name n8500x/llamacpp:jetson. Pushing to Docker Hub is a deliberate step after the image has served the network brain for a while: docker push n8500x/llamacpp-jetson:<timestamp> then :latest.

⁠Keeping the model server alive

restart/ installs llm-restart on the Jetson: a nightly restart (04:30) that frees the memory llama-server leaks over a day, a 5-minute watchdog (restarts when free memory drops under 350 MB or /health dies), and a restricted llmctl SSH user whose key may only run restart, status and updates, which is what the Pi firewall's app uses for its Restart button and its update check.