notheisz57/llamacpp repository overview

⁠llamacpp

Minimal llama.cpp server runtime for the NVIDIA GB10 (DGX Spark):

• Azure Linux base image
• Statically linked CUDA 13 build of llama-server
• Entrypoint contract for parameter-only model presets/overlays
• Weights are downloaded on first run, cached under /models (bind-mount to persist across containers).

The llamacpp- images are presets/overlays on this image. (see their descriptions for one-line run commands)

Direct use with an external/custom GGUF:

docker run -it --rm --gpus all -p 8080:8080 -v /var/cache/llama-models:/models notheisz57/llamacpp:latest -m /models/path/to/model.gguf

⁠Tags

b<llama.cpp release>_cuda (pinned), cuda13, latest.

⁠Notes

llama.cpp is built with LLAMA_CURL=OFF; downloading occurs at entrypoint, prior to llama-server call (using distro curl).

For gated Hugging Face repos, the entrypoint script accepts an ENV; include it in the run command: -e HF_TOKEN=...

On aarch64, defaults include DirectIO model loading (--load-mode dio) and CUDA unified memory (can be disabled via -e GGML_CUDA_ENABLE_UNIFIED_MEMORY=off).

The OpenAI-compatible API listens on port 8080 (/v1/..., /health).