techbysaurabh/llm-host repository overview

⁠llm-host — gateway image

Flask gateway exposing Whisper STT (faster-whisper) and Kokoro TTS at one OpenAI-compatible URL, and proxying an external LLM (vLLM / sglang). The LLM is not bundled — point the gateway at your model server.

⁠Quick start

docker run --gpus all -p 5001:5001 \
  -e VLLM_URL=http://your-vllm:8000 -e WHISPER_MODEL=large-v3 \
  techbysaurabh/llm-host:latest

Full stack (gateway + vLLM) via compose is in the repo's docker-compose.yml.

⁠Endpoints (all OpenAI-compatible)

• POST /v1/chat/completions — proxied to your LLM backend
• POST /v1/audio/transcriptions · /v1/audio/translations — Whisper
• POST /v1/audio/speech — Kokoro TTS
• GET /health · GET /v1/models

⁠Configuration

⁠Requirements

• NVIDIA GPU + nvidia-container-toolkit (cuDNN CUDA base; STT + TTS run on GPU)

Source: https://github.com/techbysaurabh/llm-host⁠ · pip install llm-host

MIT License