nullableeth/whisperx-wyoming repository overview

⁠WhisperX Wyoming - High-Accuracy Speech-to-Text

Wyoming protocol server for WhisperX with superior accuracy and speed. Integrates seamlessly with Home Assistant for voice assistants.

⁠What This Is

Speech-to-text server using WhisperX with word-level timestamps, voice activity detection, and improved accuracy over standard Whisper implementations. Up to 3-4x faster than faster-whisper after warm-up.

⁠Features

• ✅ Superior accuracy - 9.99% WER vs 11.98% for faster-whisper
• 🚀 Fast inference - 0.4-0.6s transcription after warm-up
• 🎯 Word-level timestamps - Precise alignment
• 🔊 VAD built-in - Voice activity detection filters noise
• 🌍 Multi-language - Supports all Whisper languages
• 🏠 Home Assistant ready - Wyoming protocol integration

⁠Quick Start

GPU mode (recommended):

docker run -d \
  --name whisperx \
  --gpus all \
  -p 10300:10300 \
  --restart unless-stopped \
  nullableeth/whisperx-wyoming:latest

CPU mode:

docker run -d \
  --name whisperx \
  -p 10300:10300 \
  -e WHISPER_DEVICE=cpu \
  -e WHISPER_COMPUTE_TYPE=int8 \
  --restart unless-stopped \
  nullableeth/whisperx-wyoming:latest

⁠Configuration

⁠Environment Variables (Recommended)

Configure via environment variables - no need to override the command:

⁠Docker Compose (Recommended)

services:
  whisperx:
    image: nullableeth/whisperx-wyoming:latest
    container_name: whisperx
    restart: unless-stopped
    ports:
      - "10300:10300"
    environment:
      WHISPER_MODEL: "base"          # tiny, base, small, medium, large-v2
      WHISPER_LANGUAGE: "en"         # Language code
      WHISPER_DEVICE: "cuda"         # cuda or cpu
      WHISPER_COMPUTE_TYPE: "float16" # float16 or int8
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

⁠Command-Line Arguments (Alternative)

You can also use command-line arguments (env vars take precedence if both are set):

docker run -d \
  --name whisperx \
  --gpus all \
  -p 10300:10300 \
  nullableeth/whisperx-wyoming:latest \
  --model medium \
  --language en \
  --device cuda \
  --compute-type float16

⁠Home Assistant Integration

1. Settings → Devices & Services → Add Integration
2. Search "Wyoming Protocol"
3. Host: <docker-host-ip>, Port: 10300
4. Configure voice assistant to use WhisperX for speech-to-text

⁠Available Models

Choose based on your accuracy vs speed needs:

Recommendation: Start with base, upgrade to small or medium if accuracy isn't sufficient.

⁠Supported Languages

Full language support via ISO 639-1 codes:

50+ languages total. Full list: https://github.com/openai/whisper#available-models-and-languages⁠

Set via WHISPER_LANGUAGE environment variable or --language flag.

⁠Compute Types

Recommendation: Use float16 for GPU, int8 for CPU.

⁠Performance

GPU Mode (RTX 4070 + base model):

• First transcription: ~2.5s (includes VAD model loading)
• Subsequent: ~0.4-0.6s per utterance
• VRAM usage: ~2-3GB

GPU Mode (RTX 4070 + medium model):

• First transcription: ~3s
• Subsequent: ~0.8-1.2s per utterance
• VRAM usage: ~5GB

CPU Mode (Intel i7):

• First transcription: ~8-12s
• Subsequent: ~3-5s per utterance
• RAM usage: ~4GB

Comparison to faster-whisper:

• ✅ 3-4x faster on subsequent transcriptions
• ✅ Better accuracy - correctly handles technical terms, proper nouns
• ⚠️ 2s slower on first transcription (VAD initialization)

⁠Example Configurations

⁠Maximum Accuracy (Large GPU)

environment:
  WHISPER_MODEL: "large-v3"
  WHISPER_LANGUAGE: "en"
  WHISPER_DEVICE: "cuda"
  WHISPER_COMPUTE_TYPE: "float16"

Requires ~10GB VRAM

⁠Balanced Performance (Recommended)

environment:
  WHISPER_MODEL: "base"
  WHISPER_LANGUAGE: "en"
  WHISPER_DEVICE: "cuda"
  WHISPER_COMPUTE_TYPE: "float16"

Requires ~2GB VRAM

⁠CPU-Only Mode

environment:
  WHISPER_MODEL: "base"
  WHISPER_LANGUAGE: "en"
  WHISPER_DEVICE: "cpu"
  WHISPER_COMPUTE_TYPE: "int8"
# Remove deploy.resources section

No GPU required

⁠Multi-Language

environment:
  WHISPER_MODEL: "small"
  WHISPER_LANGUAGE: "es"  # Spanish
  WHISPER_DEVICE: "cuda"
  WHISPER_COMPUTE_TYPE: "float16"

Auto-detects if language doesn't match

⁠Requirements

GPU (Recommended):

• NVIDIA GPU with CUDA 12.1+ support
• Minimum 2GB VRAM (base model)
• 5GB+ VRAM for medium/large models
• Docker with NVIDIA Container Toolkit (--gpus all)

CPU (Fallback):

• 4GB+ RAM
• Multi-core CPU recommended
• 5-10x slower than GPU

⁠Logs

Clean, timestamped logs for easy debugging:

2026-03-20 12:00:00 - INFO - Loading WhisperX base on cuda...
2026-03-20 12:00:02 - INFO - ✓ Model loaded
2026-03-20 12:00:02 - INFO - Starting server on tcp://0.0.0.0:10300
2026-03-20 12:00:15 - INFO - Transcribing 3.98s of audio...
2026-03-20 12:00:15 - INFO - ✓ Transcribed in 0.44s: 'turn on the living room lights'

⁠Troubleshooting

⁠Slow first transcription

Normal behavior! VAD model loads on first use (~2s overhead). All subsequent transcriptions are fast.

⁠GPU not detected

# Verify NVIDIA Docker support
docker run --rm --gpus all nvidia/cuda:12.1.0-base nvidia-smi

# Check container has GPU access
docker exec whisperx nvidia-smi

# Verify CUDA in container
docker exec whisperx python3 -c "import torch; print('CUDA:', torch.cuda.is_available())"

⁠Poor accuracy

• Try larger model: Upgrade from base → small → medium
• Check language: Set WHISPER_LANGUAGE to match spoken language
• Audio quality: Ensure clean audio input (16kHz recommended)
• Microphone levels: Verify levels in Home Assistant

⁠Empty transcriptions

• Audio too short: Minimum ~0.5s needed
• Microphone muted: Check HA microphone settings
• Wyoming config: Verify integration points to correct port

⁠High VRAM usage

• Use smaller model: Switch from medium → small → base
• Use int8: Set WHISPER_COMPUTE_TYPE=int8
• CPU mode: Set WHISPER_DEVICE=cpu (slower but no VRAM)

⁠Container won't start

# Check logs
docker logs whisperx

# Common issues:
# 1. Port already in use → change port
# 2. No GPU access → add --gpus all
# 3. CUDA version mismatch → update nvidia-docker

⁠Comparison to Alternatives

vs faster-whisper:

• ✅ WhisperX: 3-4x faster after warm-up
• ✅ WhisperX: Better accuracy (9.99% vs 11.98% WER)
• ⚠️ faster-whisper: Faster cold start (~500ms less)

vs OpenAI Whisper:

• ✅ WhisperX: Much faster (batched processing)
• ✅ WhisperX: Word-level timestamps
• ✅ WhisperX: Built-in VAD

vs Vosk:

• ✅ WhisperX: Better accuracy
• ✅ Vosk: Lower resource usage (~500MB RAM)
• ✅ WhisperX: Better with accents/dialects

⁠Known Issues

• First transcription takes ~2s extra for VAD initialization (normal)
• Lightning checkpoint upgrade message on startup (harmless, can be ignored)
• Requires GPU for optimal performance (CPU works but 5-10x slower)

⁠Benchmark Results

From community testing on LibriSpeech dataset:

Lower is better for WER/CER, latency varies by audio length

⁠Building From Source

git clone <your-repo>
cd whisperx-wyoming
docker build -t whisperx-wyoming:latest .

⁠Links

• WhisperX GitHub⁠
• Wyoming Protocol⁠
• Home Assistant Voice⁠
• Benchmark Source⁠

⁠License

• WhisperX: MIT License
• Whisper models: Apache 2.0
• Wyoming Protocol: MIT License

⁠Support

For issues specific to this container, please open an issue on GitHub. For WhisperX questions, see the WhisperX documentation⁠.