sinups/self-hosted-stt-gateway repository overview

⁠self-hosted-stt

Self-hosted, OpenAI-compatible speech-to-text. Point any OpenAI client at it and call audio.transcriptions.create(...). Runs locally on your GPU — no data leaves your server.

• 🎯 OpenAI-compatible POST /v1/audio/transcriptions — drop-in replacement.
• ⚡ Whisper turbo (FP16, vLLM) — ~real-time, great for in-call transcription.
• 🎚️ Model switching per request — turbo (fast) or large-v3 (accurate).
• 🔇 No phantom transcripts — Silero VAD + loudness gate drop silence, noise and quiet background voices; anti-hallucination filters drop ghost phrases and wrong-language output.
• 🔒 Token auth, language lock (e.g. Russian-only), all via env.

client ──▶ gateway (auth · VAD · loudness · language · filters · routing)
                 ├──▶ whisper-vllm     (turbo, FP16, GPU)
                 └──▶ faster-whisper   (large-v3, GPU)

⁠Hardware requirements

The turbo model runs on vLLM, which requires an NVIDIA GPU (CUDA). The gateway itself is CPU-only. Plan for the GPU.

• VRAM: FP16 turbo weights are ~1.6 GB; with the 30 s KV-cache and vLLM overhead, 8 GB is comfortable. On a tiny GPU lower VLLM_GPU_MEM (e.g. 0.30) and/or drop the faster-whisper service.
• Running both turbo + large-v3 on one GPU needs ~10–16 GB.
• No NVIDIA GPU → no turbo. You can still run large-v3 (faster-whisper) on CPU, but it is not real-time (several× slower than audio) — fine for batch/offline subtitles, not for live calls.

⁠Cloud / where it runs

⁠Software

• Docker⁠ + NVIDIA Container Toolkit⁠.
• Check your GPU's max CUDA with nvidia-smi (top-right). Driver < 570 or ≥ 580: see Troubleshooting⁠.

Verify the GPU is visible to Docker:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

⁠Quickstart (3 commands)

git clone https://github.com/sinups/self-hosted-stt.git
cd self-hosted-stt
cp .env.example .env          # then edit STT_TOKEN (openssl rand -hex 24)
docker compose up -d

First start downloads the model weights (~1.6 GB turbo, ~3 GB large-v3) — give it a couple of minutes. Watch progress:

docker compose logs -f gateway whisper-vllm

⁠Faster path — pre-built gateway image

Skip building the gateway by using the published image from Docker Hub (sinups/self-hosted-stt-gateway⁠):

cp .env.example .env          # set STT_TOKEN
docker compose -f docker-compose.images.yml up -d

(The small whisper-vllm layer is still built locally — it's quick.)

Test it:

curl -s http://localhost:8080/v1/audio/transcriptions \
  -H "Authorization: Bearer $STT_TOKEN" \
  -F [email protected] -F model=turbo
# -> {"text":"..."}   (or {"text":""} for silence/noise/quiet background)

⁠Use it (OpenAI clients)

Python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="YOUR_STT_TOKEN")
r = client.audio.transcriptions.create(
        model="turbo", file=open("sample.wav", "rb"))
print(r.text)

Node

import OpenAI from "openai";
import fs from "fs";
const client = new OpenAI({ baseURL: "http://localhost:8080/v1", apiKey: "YOUR_STT_TOKEN" });
const r = await client.audio.transcriptions.create({
  model: "turbo", file: fs.createReadStream("sample.wav") });
console.log(r.text);

⁠Request fields

Response headers expose gate diagnostics: x-stt-gate (ok/no_speech/too_quiet), x-stt-dbfs, x-stt-speech-ms, x-stt-provider.

⁠Configuration (.env)

After editing .env: docker compose up -d.

⁠Russian-only example

STT_FORCE_LANGUAGE=ru

Guarantees Russian output — no English/foreign phantom transcripts.

⁠Tuning for in-call transcription

• Send 5–15 s chunks, ideally with 1–2 s overlap or cut on speech pauses (hard fixed cuts split words/names across chunks).
• Prefer one audio stream per participant (Whisper has no diarization).
• Calibrate MIN_SPEECH_DBFS from the x-stt-dbfs header on real call audio: set it between your speaker's level and the background level.

⁠Troubleshooting

⁠CI/CD (GitHub Actions → Docker Hub)

.github/workflows/docker.yml builds and pushes the gateway image to Docker Hub automatically:

One-time setup — repo secrets (Settings → Secrets and variables → Actions):

Or via CLI:

gh secret set DOCKERHUB_USERNAME --body "sinups"
gh secret set DOCKERHUB_TOKEN    --body "dckr_pat_xxx"

⁠Releasing a version

git tag v0.2.0
git push origin v0.2.0      # CI builds & pushes sinups/self-hosted-stt-gateway:0.2.0

Then bump the tag in docker-compose.images.yml (or keep :latest).

⁠License

MIT