synonexthub/llama.cpp repository overview

Why This Image Exists (The Fixes)

Most public CUDA 12.4 images trigger fatal errors when encountering cutting-edge model architectures like Gemma 4 or crash due to strict runtime environment isolation. This build permanently resolves those issues:

1. Gemma 4 Native Support: Built from the latest upstream sources to fix the unknown model architecture: 'gemma4' or missing metadata key exceptions.
2. Linker & Stub Isolation Fix: Resolved the common libcuda.so.1 compile-time and runtime linkage deadlock inside isolated Docker sandboxes.
3. OpenMP Parallelization Guard: Hard-baked the missing GNU OpenMP library (libgomp.so.1) directly into the slim runtime environment to guarantee faultless multi-threaded CPU-GPU synchronization.
4. Pure Ampere (Ada Lovelace) Architecture Lock (Compute Arch 86 (89)): Stripped away gigabytes of legacy and enterprise code ballast (Pascal, Hopper, etc.). This makes the image incredibly lightweight, blazing fast to pull, and perfectly optimized for consumer chips like the NVIDIA GeForce RTX 30xx (40xx, 6000 Ada, L4).

⁠Quick Start Configuration

To run this image seamlessly on your local hardware with full GPU acceleration, use the following docker-compose.yaml setup:

services:
  gemma-server:
    image: synonexthub/llama.cpp:server-cuda-12.4.1-ampere
    container_name: llama-ampere-server
    restart: unless-stopped
    ports:
      - "8084:8080"
    volumes:
      - /opt/ai-stack/models:/models:ro
    environment:
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - "-m"
      - "/models/Gemma-4-E4B-IT-ABLITERATED-UNCENSORED-PHILADELPHIA-CLASS-i1.Q4_K_M.gguf"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
      - "--ctx-size"
      - "15872"
      - "--parallel"
      - "1"
      - "--batch-size"
      - "512"
      - "--ubatch-size"
      - "256"
      - "--flash-attn"
      - "on"
      - "--n-gpu-layers"
      - "42"

⁠Hardware Verification

After starting the container, verify native GPU execution and full VRAM allocation via:

docker logs llama-ampere-server
nvidia-smi

Maintained by synonexthub. Built for performance. Dedicated to the open-source community.