Docker image to run koboldcpp with GPU support
4.7K
*** THIS IS NOT OFFICIAL *** I'm not associated with LostRuins/0cc4m, if you have any problems with it feel free to message me at https://sh.itjust.works/u/noneabove1182, do not bother them on their own github please :)
Runs kobolcpp Lite web UI hosted on port 80 (which I remap in my docker-compose to 4040), and sets the koboldcpp.py file as the entrypoint so you can pass any args to it you'd like, I have my default ones below, please edit to fit your setup!
I use the following docker-compose.yml with a .env file:
docker-compose.yml:
services:
koboldcpp:
container_name: koboldcpp
restart: always
volumes:
- '${MODEL_DIR}:/app/models'
ports:
- '4040:80'
image: 'noneabove1182/koboldcpp-gpu:latest'
ulimits:
memlock: -1
mem_limit: 30gb
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
command:
[
"python3",
"koboldcpp.py",
"--model",
"/app/models/${MODEL}",
"--port",
"80",
"--threads",
"1",
"--usemlock",
"--usecublas",
"lowvram",
"--gpulayers",
"60"
]
.env:
MODEL_DIR=/home/user/models/
MODEL=WizardCoder-15B-1.0.ggmlv3.q5_1.bin
If you have any issues while updating, remove your volumes:
docker volume rm koboldcpp_kobold
Content type
Image
Digest
sha256:19fca5598…
Size
5.5 GB
Last updated
almost 2 years ago
docker pull noneabove1182/koboldcpp-gpu