andspace/whisper-trt repository overview

⁠Whisper TensorRT-LLM - Ultra-Fast Speech-to-Text Docker Images

⁠🎯 Overview

GPU-accelerated speech recognition ecosystem for OpenAI Whisper models using TensorRT-LLM. Delivers industry-leading performance for real-time speech-to-text applications with NVIDIA GPU acceleration.

Architecture: Built on top of andspace/trt-llm:base-12.8 which provides the foundational TensorRT-LLM environment with CUDA 12.8 support.

⁠📦 Available Images

⁠🔧 Compiler Image - compiler

Model Compilation & Development Environment

Foundation:

• 🏗️ Built on: TensorRT-LLM dev environment
• 🔧 Compilation: Full toolchain for model optimization
• 🤖 Automated: Built-in compilation script with smart caching

Whisper-Specific Assets:

• 🎯 Tokenizers: multilingual.tiktoken, gpt2.tiktoken
• 🎵 Audio Assets: mel_filters.npz, sample audio files
• 📊 Model Files: Pre-downloaded Whisper assets
• 🔄 Smart Caching: Persistent model cache across compilations

ML & Audio Libraries:

• 🤖 Whisper: OpenAI Whisper, KaldiAlign
• 📊 Analysis: Datasets 3.1.0, evaluation metrics, ROUGE scoring
• 🎵 Audio: Advanced AV codec support
• 📈 Visualization: Pandas, Matplotlib
• ⚡ Async: Janus framework for concurrent processing

Best for: Model compilation, TensorRT engine building, performance optimization

⁠🏷️ Advanced Tagging Strategy

⁠🎯 Latest Tag

latest → compiler  # Points to production-ready image

⁠📋 All Available Tags

Compiler Image:

andspace/whisper-trt:1.2-compiler-12.8        # Full version
andspace/whisper-trt:compiler-12.8            # Dependency version
andspace/whisper-trt:compiler                  # Type tag
andspace/whisper-trt:latest                    # Latest (→ compiler)

⁠🔄 Version Format

Full version: {release}-{type}-{cuda}
Dependency version: {type}-{cuda}
    │       │       └────────── CUDA version (12.8)
    │       └────────────────── Image type (compiler)
    └────────────────────────── Release version (1.2)

⁠🛠️ Model Compilation Workflow

⁠🚀 Automated Compilation Script

The compiler image includes an automated compilation script (compile-whisper-model.sh) that handles the complete model compilation pipeline:

Compilation Steps:

1. 📥 Model Download: Automatic download with smart caching
2. 🔄 Checkpoint Conversion: Convert to TensorRT-LLM format
3. 🏗️ Engine Building: Build optimized TensorRT engines (encoder + decoder)
4. 🧪 Testing: Validate compiled engines with test audio
5. 📊 Reporting: Generate detailed compilation report

⁠📋 Compilation Options

⁠💾 Smart Caching System

• 🔄 Shared Cache: Models cached in /models/downloaded/
• ⚡ Fast Rebuilds: Skip downloads for cached models
• 🎯 Precision Variants: Multiple precisions share same base model
• 📂 Persistent Storage: Cache survives container restarts

⁠📊 Output Structure

/models/
├── downloaded/                    🔄 Shared model cache
│   └── {model_name}.pt
└── {model_name}_{precision}/      🎯 Compiled model
    ├── encoder/
    │   ├── config.json
    │   ├── rank0.safetensors
    │   └── rank0.engine           ✅ TensorRT engine
    ├── decoder/
    │   ├── config.json
    │   ├── rank0.safetensors
    │   └── rank0.engine           ✅ TensorRT engine
    └── model_info.json            📋 Compilation metadata

⁠🎵 Supported Models & Features

⁠📢 Whisper Models

⁠⚡ Precision Support

⁠🎯 Core Capabilities

• 🔄 Automated Compilation: One-command model compilation
• 🌍 Multi-language: Support via multilingual tokenizer
• 📊 Batch Processing: Configurable batch sizes for throughput
• 🎚️ Audio Preprocessing: Advanced mel-scale filtering
• 🛡️ Error Handling: Comprehensive logging and recovery
• 🚀 Development: Complete toolchain for optimization
• 🧪 Testing: Built-in engine validation and benchmarking

⁠🚀 Quick Start

⁠🐳 Pull Image

# Model compilation & development (recommended)
docker pull andspace/whisper-trt:latest

# Version pinning with dependencies only
docker pull andspace/whisper-trt:compiler-12.8

# Full version pinning
docker pull andspace/whisper-trt:1.2-compiler-12.8

⁠⚡ Compile Models

# Basic compilation (large-v3, float16)
docker run --gpus all -v $(pwd)/models:/models \
  andspace/whisper-trt:latest \
  compile-whisper-model.sh

# Custom model and precision
docker run --gpus all -v $(pwd)/models:/models \
  andspace/whisper-trt:latest \
  compile-whisper-model.sh \
  --model small --precision float16 --batch_size 4

# High-throughput configuration
docker run --gpus all -v $(pwd)/models:/models \
  andspace/whisper-trt:latest \
  compile-whisper-model.sh \
  --model medium --precision float16 \
  --batch_size 16 --beam_width 8

⁠📋 Compilation Examples

Development Setup:

# Compile tiny model for quick testing
docker run --gpus all -v $(pwd)/models:/models \
  andspace/whisper-trt:latest \
  compile-whisper-model.sh \
  --model tiny --precision float16

Production Setup:

# Compile large-v3 for production use
docker run --gpus all -v $(pwd)/models:/models \
  andspace/whisper-trt:latest \
  compile-whisper-model.sh \
  --model large-v3 --precision float16 \
  --batch_size 8 --beam_width 4

⁠💻 System Requirements

⁠🔧 GPU Memory Requirements

⁠📊 Performance Optimizations

⁠🎯 Use Cases

• 🎙️ Real-time transcription services
• 🎧 Podcast processing and indexing
• 💼 Meeting transcription and minutes
• 🤖 Voice assistants and chatbots
• ♿ Content accessibility solutions
• 🌍 Multilingual speech processing
• 🏭 High-throughput ASR pipelines
• 🔬 Research & development environments