mannixita/xollama repository overview

⁠xollama

xollama is a soft fork of Ollama⁠. It carries fixes upstream has not merged yet (a thinking-token budget, /api/tokenize, parser and tool-call fixes), serves GGUF models on the opencoti-llamafile⁠ engine where that engine is tested, and lets a model carry its own settings: KV cache types, concurrent slots, engine, device.

Same API, same clients, same models as Ollama. Two differences to know before you start: it listens on 22434, and its listen address is set with XOLLAMA_HOST, never OLLAMA_HOST.

• Source: https://github.com/mann1x/xollama⁠
• Full Docker guide: docs/xollama/docker.mdx⁠
• Also published as ghcr.io/mann1x/xollama, with the same tags.

xollama is not affiliated with or endorsed by Ollama.

⁠Supported tags

A build moves latest or dev, never both. If latest does not exist yet, no full release has been published as an image: use dev or a version tag.

Architecture: linux/amd64 only. No arm64 image and no ROCm build yet.

⁠What is inside

• /usr/bin/xollama — server and CLI. It is the entrypoint; the default command is serve.
• /usr/lib/ollama — stock llama.cpp (llama-server) with upstream Ollama's CUDA v12, CUDA v13 and Vulkan backends, and the opencoti engine with a CUDA 13 payload plus a CUDA 12 payload for Volta cards.
• /usr/lib/ollama/PAYLOAD — which bytes this image carries: xollama version, runtime release and sha256, upstream tarballs, engine tag and revision.
• Base: ubuntu:24.04. Runs as root. Exposes 22434.

Nothing native is compiled when the image is built: every piece is a published artifact pinned by sha256 in the repository, and CI checks that the server answers on the published port before pushing.

⁠Quick start

⁠docker run

With an NVIDIA GPU (set up GPU passthrough⁠ first):

docker run -d --gpus=all -v xollama:/root/.ollama -p 22434:22434 --name xollama mannixita/xollama

CPU only:

docker run -d -v xollama:/root/.ollama -p 22434:22434 --name xollama mannixita/xollama

Until the first full release moves latest, add :dev to the image name. Add --restart unless-stopped to keep it running across reboots. Then:

curl http://localhost:22434/api/version
docker exec -it xollama xollama run qwen3.6

⁠docker compose

services:
  xollama:
    image: mannixita/xollama:latest
    container_name: xollama
    restart: unless-stopped
    ports:
      - "22434:22434"
    volumes:
      - xollama:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: 10m
      # XOLLAMA_API_KEY: change-me-to-a-long-random-key
      # XOLLAMA_PARALLEL: 2
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  xollama:

docker compose up -d
docker compose exec xollama xollama run qwen3.6

⁠Parameters

⁠Ports

Inside the container the server binds XOLLAMA_HOST=0.0.0.0:22434. To move it, set -e XOLLAMA_HOST=0.0.0.0:<port> and publish that port. Setting OLLAMA_HOST does nothing: it is not read for the listen address.

⁠Volumes

File ownership. The container runs as root, but files it creates in the store are handed to the owner (numeric uid:gid) of the directory they go into, when that owner is not root. Mount a directory your account owns and the models it pulls belong to your account on the host.

Do not share a store with a running Ollama. One model store with two writers and one GPU with two schedulers is not supported, whichever ports and containers they use. Give xollama its own directory, or stop the other server.

⁠Environment variables

Every OLLAMA_ variable can also be written XOLLAMA_, and that spelling wins when both are set. The listen address is the exception: XOLLAMA_HOST only.

Server

Security

The key guards connections to this server only. It is not an ollama.com key, and OLLAMA_API_KEY is not read for it. Over plain HTTP it travels in clear text, so put a TLS reverse proxy in front of any port other machines use. The CLI inside the container sends the same variable, so docker exec keeps working.

Engine

KV cache

Concurrent requests

Council

At startup the server logs every variable with its current value, in the server config line of docker logs xollama.

⁠GPU support

⁠NVIDIA

A container sees an NVIDIA GPU only when Docker passes it in. On the host:

1. The NVIDIA driver, working: nvidia-smi lists the GPUs. 580 or newer for the CUDA 13 payload, 570 or newer for Volta's CUDA 12 payload.

2. The NVIDIA Container Toolkit⁠: add NVIDIA's package repository as that guide describes, then sudo apt-get install -y nvidia-container-toolkit (Debian/Ubuntu) or sudo dnf install -y nvidia-container-toolkit (Fedora/RHEL family).

3. Docker configured for it:

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker
   

   Copy

Check it; this must list your GPUs:

docker run --rm --gpus=all ubuntu nvidia-smi

Then start xollama with --gpus=all (or --gpus '"device=0,1"' for some of them). The image sets NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=compute,utility. Docker Desktop on Windows (WSL 2) needs only the NVIDIA Windows driver.

could not select device driver "" with capabilities: [[gpu]] means step 2 or 3 is missing.

Which engine serves a load depends on the card. With the currently pinned engine (opencoti build 2609271900001):

The engine chosen for each load, and why, is in docker logs xollama.

⁠Vulkan

Upstream's Vulkan backend and the Vulkan loader are in the image, as in upstream's own image; pass the devices with --device /dev/dri. OLLAMA_VULKAN=0 turns Vulkan off and GGML_VK_VISIBLE_DEVICES selects devices. The pinned engine has no Vulkan payload, so Vulkan loads are served by stock llama.cpp.

⁠Configuring slots and KV cache

On the opencoti engine a model's KV cache is one pool: num_ctx × the slots it loads with. Those slots are the model's own slots.live if it sets one, else XOLLAMA_PARALLEL, else 1. OLLAMA_NUM_PARALLEL is not read on opencoti. With nothing set, a model with no context of its own on a 24 GB card loads a 32k × 1 pool.

A request that asks for no particular window is guaranteed the whole pool, so by default requests to one model take turns, as on Ollama. There are two ways to run them side by side:

• Clients book windows. A /api/chat request with placement.num_ctx gets that window, and up to XOLLAMA_MAX_PARALLEL of them share the pool while it has room. A model loaded with num_ctx 262144 serves one 256k conversation, two of 128k or four of 64k from the same load. The X-Context-Window response header says what was granted.
• The operator sizes the pool. XOLLAMA_PARALLEL=4 allocates four contexts at load, and four requests that ask for no window each get a whole one. That is Ollama's OLLAMA_NUM_PARALLEL=4, with the same memory cost.

The OpenAI and Anthropic routes carry no placement, so their requests take the whole pool unless it is sized with XOLLAMA_PARALLEL. Do not vary options.num_ctx per request: it sizes the load, and a new value reloads the model.

Example — a 64k × 2 pool with a q8_0 cache:

docker run -d --name xollama --gpus all -p 22434:22434 \
  -v xollama:/root/.ollama \
  -e XOLLAMA_PARALLEL=2 \
  -e OLLAMA_CONTEXT_LENGTH=65536 \
  -e OLLAMA_FLASH_ATTENTION=1 \
  -e OLLAMA_KV_CACHE_TYPE=q8_0 \
  mannixita/xollama:latest

⁠KV cache types

Four settings: keys (XOLLAMA_K_CACHE_TYPE), values (XOLLAMA_V_CACHE_TYPE) and, for a sliding-window model such as Gemma-4, the two halves of its short-window ring (XOLLAMA_K_CACHE_TYPE_SWA, XOLLAMA_V_CACHE_TYPE_SWA). OLLAMA_KV_CACHE_TYPE sets keys and values together.

Presets (keys / values):

On a V100 start with q8_0 / q8_0.

Values

Combinations

• Two plain types may differ (q8_0 / q4_0); two kvarn widths may differ (kvarn4 / kvarn3).
• Do not mix a kvarn width with a plain type: the engine silently promotes the plain half to the same kvarn width.
• A ring needs kvarn on both keys and values. Set both ring halves, both kvarn or both plain. They may differ from each other (q8_0 / q4_0).
• Quantised values need flash attention: set OLLAMA_FLASH_ATTENTION=1.
• On CUDA, q6_0 mixed with another type crashes in q4_0/q6_0, q6_0/q8_0 and q6_0/q4_1. Use q6_0 / q6_0 or skip q6_0.
• Put kvarn and q6_0 in the XOLLAMA_* variables. Keep OLLAMA_KV_CACHE_TYPE to f16, q8_0 or q4_0: it is the fallback when stock llama.cpp serves a device the engine does not cover.

Example, balanced for a sliding-window model:

docker run -d --name xollama --gpus all -p 22434:22434 \
  -v xollama:/root/.ollama \
  -e OLLAMA_FLASH_ATTENTION=1 \
  -e OLLAMA_KV_CACHE_TYPE=q8_0 \
  -e XOLLAMA_K_CACHE_TYPE=kvarn4 -e XOLLAMA_V_CACHE_TYPE=kvarn4 \
  -e XOLLAMA_K_CACHE_TYPE_SWA=q8_0 -e XOLLAMA_V_CACHE_TYPE_SWA=q8_0 \
  mannixita/xollama:dev

All the rules: KV cache types — values and combinations⁠.

The environment sets server-wide defaults. A model can carry its own KV cache types, slots, engine and device pin, which win over the environment, set with xollama tweak model inside the container:

docker exec -it xollama xollama tweak model qwen3.6
docker exec -it xollama xollama tweak model qwen3.6 --kv-k=q8_0 --kv-v=q8_0
docker exec -it xollama xollama tweak model qwen3.6 --slots-live=2

The settings are stored in the model, in the volume. Details: slots⁠ · KV cache types⁠ · tweak⁠.

⁠Updating

Models and settings live in the volume, so updating is pull and recreate:

docker pull mannixita/xollama:latest
docker rm -f xollama
docker run -d --name xollama --gpus all -p 22434:22434 \
  -v xollama:/root/.ollama --restart unless-stopped mannixita/xollama:latest

With Compose: docker compose pull && docker compose up -d.

To stay on one build, use a <version> tag. curl http://localhost:22434/api/xollama reports the version a running server was built as.

⁠Where the image comes from

The image is assembled by the docker-release⁠ workflow on GitHub-hosted runners, from Dockerfile.xollama⁠ and scripts/docker-assemble.sh⁠:

To see what an image holds:

docker run --rm --entrypoint cat mannixita/xollama:latest /usr/lib/ollama/PAYLOAD

Details: docs/features/docker-release.md⁠.

⁠Support and links

• Source and issues: https://github.com/mann1x/xollama⁠
• Documentation: https://github.com/mann1x/xollama/tree/main/docs/xollama⁠
• The engine: https://huggingface.co/ManniX-ITA/opencoti-llamafile⁠
• Upstream Ollama: https://github.com/ollama/ollama⁠

Please report problems with this image to the xollama repository, not to Ollama.

⁠License

MIT, unchanged from upstream Ollama — see LICENSE⁠. Copyright (c) Ollama, plus xollama's contributors for the changes in the fork. The bundled engine and libraries carry their own licences; the Go dependencies' licences are in /usr/lib/ollama/GO_LICENSE.