synonexthub/prism-llama-cuda repository overview

A highly optimized, hardened, and minimalist production runtime environment for llama-server (based on the ggml/llama.cpp architecture). This image is specifically designed for asynchronous inference pipelines utilizing highly compressed models (Ternary, Q1_0, PQ2_0) and advanced inference techniques—such as Speculative Decoding and Multi-Token Prediction (MTP)—on consumer-grade GPUs (e.g., NVIDIA T400 4GB or RTX 3050, 6–8 GB VRAM).

🚀 Features

1. Native Driver Injection: The image contains no proprietary graphics driver libraries. Drivers (libcuda.so.1) are dynamically injected at runtime via the host system's NVIDIA Container Toolkit.

2. Full Feature Set: Native support for CUDA Tensor Cores, Flash Attention (--flash-attn), and hardware-accelerated offloading (CPU/GPU layer splitting).

3. Zero Hard-coded Configuration: Complete portability. Credentials, file paths, and model configurations are managed exclusively via runtime arguments.

4. Pure Ampere (Ada Lovelace, T400) Architecture Lock (Compute Arch (75),86,(89)): Stripped away gigabytes of legacy and enterprise code ballast (Pascal, Hopper, etc.). This makes the image incredibly lightweight, blazing fast to pull, and perfectly optimized for consumer chips like the NVIDIA GeForce RTX 30xx (40xx, 6000 Ada, L4) or T400.

⁠Quick Start Configuration (Ampere: RTX 3050 / 8GB VRAM)

To run this image seamlessly on your local hardware with full GPU acceleration, use the following docker-compose.yaml setup:

version: "3.8"

services:
  llama-cpp:
    container_name: llama-cpp-prism
    image: synonexthub/prism-llama-cuda:ampere
    restart: unless-stopped
    runtime: nvidia
    ipc: host
    ports:
      - "8086:8080"
    environment:
      - TZ=Europe/Berlin
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    volumes:
      - /volume1/docker/models:/models:ro
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    command:
      --model /models/Ternary-Bonsai-2-27B-Abliterated-PTQ1_0-MTP-Q4_0.gguf
      --host 0.0.0.0
      --port 8080
      --cache-type-k q4_0
      --n-gpu-layers 66
      --ctx-size 32768
      --fit off
      --flash-attn on
      --jinja
      -b 512
      -ub 256
      --parallel 1
      --spec-type draft-mtp
      --spec-draft-n-max 2

⁠Hardware Verification

After starting the container, verify native GPU execution and full VRAM allocation via:

docker logs llama-cpp-prism
nvidia-smi

Maintained by synonexthub. Built for performance. Dedicated to the open-source community.