chartavellas/colibri repository overview

⁠Colibrì — frontier LLMs on hardware you already own

744B to 2.8T parameters, running locally. All five Colibrì model families in one image — NVIDIA, AMD and Intel GPUs, a web UI, and an OpenAI-compatible API.

This is a ready-to-run Docker image for Colibrì⁠, an inference engine that treats VRAM, RAM and disk as one memory hierarchy and streams mixture-of-experts weights from storage on demand. That's what lets a 744B model run on a workstation instead of a rack.

docker pull chartavellas/colibri:latest

New to this? Start with OLMoE — 4 GB of weights, 8 GB of RAM, no GPU needed. You can be talking to it in ten minutes instead of after a 372 GB download.

⁠Who this is for

You want to run a frontier-class model privately — on your own machine, with your own data, without renting an API. A GPU is welcome but not required.

This is not for you if you need production-grade throughput. Speed is bound by your disk, because experts are streamed from it. Expect a fraction of a token per second on a slow drive, and a few tokens per second on fast NVMe with a warm cache. Colibrì is explicitly a research engine — its authors publish no SLA on speed, and a hard guarantee on semantics: low memory makes it slower, never quietly less correct.

⁠What's in the image

• All five Colibrì engines, built from the matching upstream release
• CUDA 12.8 for NVIDIA, compiled for Ampere through Blackwell
• Vulkan for AMD and Intel, with the Mesa drivers included — no ROCm needed
• The Colibrì Web UI — live token metrics, VRAM/RAM/disk tier bar, and the Brain/Atlas expert visualisations
• OpenAI-compatible HTTP API
• Automatic VRAM/RAM/disk expert tiering (--auto-tier)
• vulkaninfo for checking what your machine actually exposes

⁠Supported models

Every family below is built into the image. coli reads the model's config.json and starts the right engine — you don't pick one, just point it at your weights.

None of these requires a GPU — that RAM column is the real gate, and it's far lower than the parameter counts suggest, because experts stream from disk. A GPU only ever makes it faster.

Two of the five carry no GPU code, which we verified with cuobjdump against the shipped binaries rather than inferring from build flags. OLMoE contains no GPU code upstream at all, and DeepSeek V4 Flash is a separate CPU runtime. Both run perfectly well — CPU is the normal way to run Colibrì — they simply will not touch your card.

⁠Supported GPUs

The image carries both GPU backends — CUDA for NVIDIA, Vulkan for AMD and Intel — so one tag covers all three vendors. No separate builds to pick between.

NVIDIA — CUDA

Needs a driver new enough for CUDA 12.8 (570+ is the safe floor, and is required for RTX 50-series regardless) plus the NVIDIA Container Toolkit⁠.

AMD and Intel — Vulkan

AMD support goes back to Polaris / RX 580 via Mesa's RADV driver, and Intel Arc via ANV. Both ship in the image; you need no ROCm install and no vendor toolkit. Upstream measured the Vulkan int4 expert-group primitive ~35% faster than their own ROCm/HIP path on an RX 9070, so Vulkan is not a fallback here — on AMD it is the fast path.

Pass the render device through instead of --gpus all:

docker run -d --name colibri --device /dev/dri -p 8080:8080 -v /mnt/models/GLM-5.2:/model:ro -e COLI_API_KEY="your-secret" chartavellas/colibri:latest

Vulkan acceleration applies to GLM-5.2 and Kimi K3 — the two engines with a Vulkan backend upstream. Inkling is CUDA-only, and OLMoE and DeepSeek V4 are CPU engines on any vendor.

Check what your machine offers:

docker run --rm --gpus all chartavellas/colibri:latest vulkaninfo --summary

Look at deviceType. PHYSICAL_DEVICE_TYPE_DISCRETE_GPU with your card's name is what you want on AMD or Intel. If you get no devices, check the host driver and that you passed --device /dev/dri; Colibrì still runs on CPU.

On NVIDIA, Vulkan reporting no devices is fine and expected. You will see:

ERROR: Could not get 'vkCreateInstance' ... for ICD libGLX_nvidia.so.0
ERROR: Failed to detect any valid GPUs in the current config

Nothing is wrong. NVIDIA's Vulkan ICD does not initialise inside the container, and it does not matter: CUDA is the NVIDIA path and the faster one. Vulkan is here for AMD and Intel, which have no CUDA. Check your NVIDIA GPU with nvidia-smi instead.

You should never see a software device such as llvmpipe. Mesa's software rasterizer is removed on purpose: it advertises itself as a valid Vulkan device and would run compute shaders on the CPU — slower than Colibrì's own CPU path, while looking accelerated. If one ever appears, that is a bug in this image.

The GPU is optional entirely. Colibrì runs every model on CPU alone. With no usable card, drop --gpus all / --device /dev/dri and it still works — slower, never less correct.

Storage speed is the single biggest factor in how fast this feels. NVMe is strongly preferred over SATA SSD, and a spinning disk will be painful.

⁠Weights are not included

The image ships the engines, not the weights — that's the difference between a 2 GB pull and a 1.6 TB one. Fetch whichever model you want and mount it:

# OLMoE — smallest, no GPU needed, good first run
huggingface-cli download allenai/OLMoE-1B-7B-0125-Instruct --local-dir /mnt/models/OLMoE

# GLM-5.2 — the reference model (~372 GB)
huggingface-cli download mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp --local-dir /mnt/models/GLM-5.2

# Inkling (~469 GB)
huggingface-cli download nbeerbower/Inkling-colibri-int4 --local-dir /mnt/models/Inkling

# Kimi K3 — streams straight from the original shards, no conversion (~1.6 TB)
huggingface-cli download moonshotai/Kimi-K3 --local-dir /mnt/models/Kimi-K3

OLMoE needs a conversion pass (c/tools/convert_olmoe_merged.py) and DeepSeek V4 uses the official sharded checkpoint — see the upstream docs⁠ for both.

⁠Quick start

Point /model at whichever weights you downloaded — the launcher works out which engine to start from the model's own config.json:

docker run -d --name colibri --gpus all -p 8080:8080 -v /mnt/models/GLM-5.2:/model:ro -e COLI_API_KEY="CHANGE-THIS-TO-A-STRONG-SECRET" chartavellas/colibri:latest

Then open http://localhost:8080⁠ — or http://YOUR-SERVER-IP:8080 from another machine.

That's the whole command, and it's the same for every model family. The image already starts the web UI with sensible defaults, so there's nothing to append.

First start takes a while: the engine reads the model layout and plans expert placement across VRAM, RAM and disk. Watch it with docker logs -f colibri.

⁠Using the API

The server speaks the OpenAI chat-completions protocol, so existing clients work by pointing them at it:

curl http://localhost:8080/v1/chat/completions -H "Authorization: Bearer CHANGE-THIS-TO-A-STRONG-SECRET" -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"Explain MoE routing in two sentences."}]}'

Any OpenAI SDK works the same way — set the base URL to http://localhost:8080/v1 and the API key to your COLI_API_KEY.

⁠Tuning

The defaults are --ctx 4096 --ngen 128 --temp 0.7 --topp 0.9 with --auto-tier. Override the whole command to change them:

docker run -d --name colibri --gpus all -p 8080:8080 -v /mnt/models/GLM-5.2:/model:ro -e COLI_API_KEY="your-secret" chartavellas/colibri:latest /bin/bash -lc 'exec ./coli web --model /model --host 0.0.0.0 --port 8080 --allowed-host "*" --auto-tier --ctx 16384 --ngen 512 --temp 0.7 --topp 0.9 --no-browser'

Raising --ctx costs memory; --auto-tier will rebalance placement to fit.

⁠Before a long run

Two read-only commands worth knowing:

docker exec colibri ./coli doctor --model /model

doctor is a readiness check — it tells you whether the model, memory and disk add up before you wait on a load. plan (same form) shows the placement it intends across VRAM, RAM and disk, which is the fastest way to understand why a given machine is slow.

⁠Security

Two defaults are tuned for convenience behind Docker, and both deserve a look before you expose this beyond your own network:

--allowed-host "*" accepts any Host header. Convenient behind a reverse proxy, too permissive on a public address — override the command with an explicit hostname.

COLI_API_KEY is not baked into the image and has no default. If you don't set it, think carefully about who can reach port 8080; anyone who can is talking to your model.

⁠Tags

chartavellas/colibri:latest
chartavellas/colibri:<version>-webui-cuda12.8

<version> tracks the upstream Colibrì release, e.g. 1.6.2-webui-cuda12.8. The image rebuilds automatically when upstream publishes a new release.

⁠Credits & links

Colibrì is built by JustVugg⁠ — this image only packages it.

• Engine & docs — github.com/JustVugg/colibri⁠
• Website — justvugg.github.io/colibri⁠
• Image source — github.com/Chartavellas/colibri-docker⁠

Issues with the engine belong upstream; issues with the image belong on the image repo.