llama.cpp for quantizing and inferencing large language models in C++
10K+
llama.cpp is a set of tools for inferencing large language models in C++.
Unofficial Docker image for the version downloaded 2026-09-04.
Contains:
These instructions are for Linux.
On the Docker host:
# Set the path to your downloaded Huggingface model and your desired quantization types separated by spaces
INPUT_MODEL=/path/to/your/model
QUANT_TYPES="Q4_K_M Q5_K_M Q8_0"
docker run -it --rm \
-v$INPUT_MODEL:/models/input-model \
--runtime=nvidia --gpus all agentlans/llamacpp \
./QuantizeTextModel.sh $QUANT_TYPES
If you have a directory containing training data for tuning the quantization
INPUT_MODEL=/path/to/your/model
QUANT_TYPES="Q4_K_M Q5_K_M Q8_0"
# The training data is the concatenation of all the files in this directory (non-recursive)
TRAINING_DATA=/path/to/your/training/data/
docker run -it --rm \
-v$INPUT_MODEL:/models/input-model \
-v$TRAINING_DATA:/training-data \
--runtime=nvidia --gpus all agentlans/llamacpp \
./QuantizeTextModel.sh $QUANT_TYPES
If successful, the model directory would look like this:
/path/to/your/model
├── GGUF (newly created directory containing the output files)
│ ├── imatrix.dat (if training data was provided for tuned quantization)
│ ├── model.gguf (the FP16 GGUF model)
│ ├── Q4_K_M.gguf (various quantized GGUFs)
│ ├── Q5_K_M.gguf
│ └── Q8_0.gguf
...
To see all the allowed quantization types, run this on the Docker host:
docker run -it --rm \
--runtime=nvidia --gpus all agentlans/llamacpp \
./bin/llama-quantize -h
To start the server on port 5000 on the host:
MODEL_DIR=/path/to/your/model
QUANT_FILE=Q4_K_M.gguf
docker run --rm -it \
-v $MODEL_DIR:/app/llama.cpp/models -p5000:8080 \
--runtime=nvidia --gpus all agentlans/llamacpp \
./bin/llama-server --host 0.0.0.0 --flash-attn auto \
--gpu-layers 100 \
-m models/$QUANT_FILE \
--api-key MY_SECRET_KEY
See official documentation for details.
Content type
Image
Digest
sha256:9133df6b6…
Size
5 GB
Last updated
20 days ago
docker pull agentlans/llamacpp