cslev/llamacpp-cuda-amd64 repository overview

https://github.com/cslev/llamacpp-cuda-amd64-docker⁠

⁠Release Notes - llama.cpp CUDA Docker for AMD64

⁠🚀 Overview

This release provides a production-ready Docker image for running llama.cpp with CUDA acceleration on AMD64 architectures. The image enables efficient large language model inference with GPU offloading for NVIDIA GPUs.

⁠✨ Key Features

⁠Multi-Architecture GPU Support

• A100 (Compute Capability 80) - Ampere Datacenter
• RTX 30-series (Compute Capability 86) - RTX 3080, 3090 & A6000
• RTX 40-series (Compute Capability 89) - RTX 4080, 4090 & L40
• H100 & H200 (Compute Capability 90) - Hopper Datacenter

⁠Core Capabilities

• CUDA 12.2 base image for optimal GPU performance
• Multi-stage Docker build for minimal runtime footprint
• Router Mode support with model presets via models.ini
• Multimodal support including vision models (e.g., Qwen2.5-VL) with mmproj files
• Flash attention for improved performance
• Parallel inference with configurable concurrency
• Context shifting for handling long conversations

⁠Optimized Configuration

• Up to 99 GPU layers offloading
• 16K context size support
• CURL support for remote model loading
• Exposed on port 8033 (mapped to host port 3000 in docker-compose)

⁠🐳 Docker Image

Pull from Docker Hub:

docker pull cslev/llamacpp-cuda-amd64:latest