andspace/trt-llm repository overview

⁠TensorRT-LLM Docker Images - Production-Ready NVIDIA AI Inference Platform

⁠🎯 Overview

High-performance Docker ecosystem for TensorRT-LLM development and deployment, built on NVIDIA CUDA 12.4 with Ubuntu 22.04. Optimized for GPU-accelerated AI model inference and compilation workflows.

⁠📦 Available Images

⁠🔧 Base Image - base

Runtime Environment for Production Inference

Core Stack:

• 🎯 NVIDIA CUDA: 12.4.1 Runtime
• 🔥 PyTorch: With CUDA support
• ⚡ TensorRT-LLM: 0.21 (Production Ready)
• 🐍 Python: 3.10+ with pip, setuptools, wheel

Pre-installed Libraries:

• 🧮 Scientific: NumPy, SciPy
• 🤖 AI/ML: Transformers, Tokenizers, TikToken
• 🎵 Media: FFmpeg for audio/video processing

Best for: Production inference, model serving, runtime deployment

⁠🛠️ Development Image - dev

Complete Compilation Environment

Build Environment:

• 🔨 Compilation Tools: GCC, CMake, Ninja, Git LFS
• 🌐 Parallel Computing: OpenMPI development headers
• 📂 Pre-configured Workspace: /workspace with TensorRT-LLM v0.21 source

Advanced Features:

• 🎯 GPU Optimization: CUDA architecture targeting (8.6)
• 🔧 Dependencies: MPI4py, packaging tools, typing extensions
• 📋 Ready-to-use: Cloned TensorRT-LLM repository with submodules

Best for: Model compilation, custom engine building, development workflows

⁠🏷️ Tagging Strategy

⁠🎯 Latest Tag

latest → base  # Points to most commonly used image

⁠📋 All Available Tags

Base Image:

andspace/trt-llm:1.0-base-12.4-0.21    # Full version
andspace/trt-llm:1.0-base               # Short version  
andspace/trt-llm:base                   # Type tag
andspace/trt-llm:latest                 # Latest (→ base)

Development Image:

andspace/trt-llm:1.0-dev-12.4-0.21     # Full version
andspace/trt-llm:1.0-dev                # Short version
andspace/trt-llm:dev                    # Type tag

⁠🔄 Version Format

{release}-{type}-{cuda}-{tensorrt}
    │       │       │        └─ TensorRT-LLM version (major.minor)
    │       │       └────────── CUDA version (major.minor)  
    │       └────────────────── Image type (base/dev)
    └────────────────────────── Release version (1.0)

⁠🚀 Quick Start

# Runtime inference (recommended)
docker pull andspace/trt-llm:latest

# Development & compilation
docker pull andspace/trt-llm:dev

# Version pinning
docker pull andspace/trt-llm:1.0-base-12.4-0.21

⁠💻 System Requirements

⁠🎯 Use Cases

• ⚡ Large Language Model inference
• 🔧 TensorRT engine compilation
• 🚀 CUDA-accelerated AI workloads
• 🏭 Production ML deployments
• 🔬 Research environments