noneabove1182/koboldcpp-cpu repository overview

*** THIS IS NOT OFFICIAL *** I'm not associated with LostRuins/0cc4m, if you have any problems with it feel free to message me at https://sh.itjust.works/u/noneabove1182⁠, do not bother them on their own github please :)

Runs kobolcpp Lite web UI hosted on port 80 (which I remap in my docker-compose to 4040), and sets the koboldcpp.py file as the entrypoint so you can pass any args to it you'd like, I have my default ones below, please edit to fit your setup!

I use the following docker-compose.yml with a .env file:

docker-compose.yml:

services:
  koboldcpp:
    container_name: koboldcpp
    restart: always
    volumes:
      - '${MODEL_DIR}:/app/models'
    ports:
      - '4040:80'
    image: 'noneabove1182/koboldcpp-cpu:latest'
    ulimits:
      memlock: -1
    mem_limit: 30gb
    command:
      [
        "python3",
        "koboldcpp.py",
        "--model",
        "/app/models/${MODEL}",
        "--port",
        "80",
        "--threads",
        "6",
        "--usemlock",
      ]

.env:

MODEL_DIR=/home/user/models/
MODEL=WizardCoder-15B-1.0.ggmlv3.q5_1.bin

If you have any issues while updating, remove your volumes:

docker volume rm koboldcpp_kobold