emmaylots/prismml-llama-bonsai repository overview

⁠prismml-llama-bonsai

Ships the pllama command-line tool, so commands below read pllama pull, pllama run and so on.

Run PrismML's ternary Bonsai 2 LLMs, or any GGUF from Hugging Face, with an Ollama-style CLI and an OpenAI-compatible API.

Bonsai 2 packs its weights into about 1.75–2.13 bits each, so Bonsai 2 27B is 6–7 GB instead of ~54 GB. It fits on an 8 GB GPU with vision, tool calling and thinking intact. pllama wraps PrismML's llama.cpp fork⁠, which is needed for the PQ2_0 / PTQ1_0 packings, in llama-server router mode. One endpoint serves many models, loads them on demand, evicts the least recently used one, and keeps models in memory forever or unloads them when idle.

⁠Tags

0.2.0 bundles llama.cpp fork release prism-b10777-e3de715.

⁠Quick start (NVIDIA)

Requirements: NVIDIA driver ≥ 570 (the image is built on CUDA 12.8), plus the NVIDIA Container Toolkit⁠.

docker run -d --name pllama --gpus all --restart unless-stopped \
  -p 127.0.0.1:11435:8080 -v pllama-models:/models \
  emmaylots/prismml-llama-bonsai:cuda

docker exec pllama pllama pull bonsai2:27b       # picks the packing that fits your VRAM
docker exec -it pllama pllama run bonsai2:27b    # interactive chat

CPU only: use emmaylots/prismml-llama-bonsai:cpu and drop --gpus all.

API: http://localhost:11435/v1 (OpenAI-compatible: chat, completions, tool calls, image input). Web UI: http://localhost:11435/. Open WebUI: add an OpenAI connection with that /v1 URL.

curl http://localhost:11435/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "bonsai2:27b",
  "messages": [{"role": "user", "content": "Explain ternary weights in two sentences."}]
}'

⁠docker compose

services:
  pllama:
    image: emmaylots/prismml-llama-bonsai:cuda
    container_name: pllama
    networks: [pllama]
    restart: unless-stopped
    ports: ["127.0.0.1:11435:8080"]       # "0.0.0.0:11435:8080" to expose on your LAN (set an API key!)
    volumes: ["./models:/models"]
    ulimits: { memlock: { soft: -1, hard: -1 } }
    env_file: .env                        # HF_TOKEN=..., PLLAMA_API_KEY=..., etc.
    environment:
      PLLAMA_PRELOAD: bonsai2:27b        # pull + load at startup
      PLLAMA_KEEP_ALIVE: "-1"            # keep in VRAM forever
    deploy:
      resources:
        reservations:
          devices: [{ driver: nvidia, count: all, capabilities: [gpu] }]

networks:
  pllama:
    name: pllama                         # fixed name other projects can join (see below)

⁠Using pllama from other containers

The compose files put pllama on a Docker network with the fixed name pllama. Containers on that network reach it by name, on the container port:

http://pllama:8080/v1        # not 11435: that port only exists on the host

The 127.0.0.1:11435 host binding has no effect on this; it only controls access from the host. Containers not on the network can't resolve pllama, and host.docker.internal:11435 fails too, because the host port listens on 127.0.0.1 only.

To connect a container from another compose project, join the network as external. For example, Open WebUI:

# open-webui/docker-compose.yml
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports: ["127.0.0.1:3000:8080"]
    volumes: ["open-webui:/app/backend/data"]
    environment:
      OPENAI_API_BASE_URL: http://pllama:8080/v1
      OPENAI_API_KEY: ${PLLAMA_API_KEY:-none}   # match PLLAMA_API_KEY if you set one
      ENABLE_OLLAMA_API: "false"
    networks: [default, pllama]

volumes:
  open-webui:

networks:
  pllama:
    external: true      # start pllama first: it creates the network

From a plain docker run: docker run --network pllama .... Every container on the network can call pllama without logging in unless PLLAMA_API_KEY is set.

⁠Commands

⁠Models

Bonsai models come from prism-ml on Hugging Face⁠ (Apache 2.0).

⁠Measured on an RTX 5050 (8 GB)

q8_0 KV cache, 32K context, 233-token prompt, 128 tokens generated:

The 27B runs at full speed up to about 40K context on 8 GB.

⁠Configuration

All settings are environment variables. The defaults are tuned for a single GPU.

Sampling defaults follow each model card. Any field sent in a request overrides them.

⁠Changelog

• 0.2.0: renamed: the command is now pllama, the container and network are pllama, and settings are PLLAMA_*. Old XOLLAMA_* variables are not read; a startup warning names each one to rename. Models need no migration.
• 0.1.1: model commands accept the IDs ps prints (bonsai2/4b as well as bonsai2:4b).
• 0.1.0: first release.

⁠Security

The server has no authentication unless you set PLLAMA_API_KEY. Keep the port on 127.0.0.1 or set a key before you publish it to a network.