toolboc/nv-llama repository overview

⁠Download models into local folder then mount and run as shown:

docker run --rm -it --name llama --net=host --gpus all -v ~/src/llama.cpp/models:/models toolboc/nv-llama:r35.2.1 --run --model /models/13B/llama-13b.ggmlv3.q6_K.bin --n-predict 512 --n-gpu-layers 43 --repeat_penalty 1.0 --color --interactive-first