techbysaurabh/whisper-blaze repository overview

⁠whisper-blaze

High-throughput batched serving for Whisper large-v3⁠ on NVIDIA Hopper GPUs, with an OpenAI-compatible API and VRAM capping for shared GPUs.

⁠Quick start

docker run --gpus all -p 8000:8000 -v hf-cache:/data/hf \
  techbysaurabh/whisper-blaze:latest

curl -F [email protected] -F language=en localhost:8000/v1/audio/transcriptions

Existing OpenAI audio clients work by changing the base URL. Health: GET /health.

⁠What it does

• Dynamic cross-request batching — concurrent requests are fused into a single GPU pass instead of being decoded one chunk at a time.
• VRAM capping — -e VRAM_LIMIT_GB=24 runs in 24 GB of an 80 GB card and sizes each batch to fit, so the GPU can be shared with other workloads.
• Two long-form modes, per request — fast (default) or accurate, which stitches chunks on Whisper's own segment timestamps for materially lower word error rate at roughly half the throughput.
• Fused Hopper LayerNorm kernel — a hand-written CUDA kernel used for every LayerNorm in the model.

⁠Configuration

⁠Requirements

• NVIDIA Hopper GPU (SM90 / sm_90a)
• NVIDIA driver 550+, nvidia-container-toolkit

⁠Links

• Source: https://github.com/techbysaurabh/whisper-blaze⁠
• Python package: https://pypi.org/project/whisper-blaze/⁠

MIT License