liuzhenghua66/llm-router repository overview

⁠LLM Router

A lightweight, production-ready LLM gateway compatible with OpenAI and Anthropic APIs.

LLM Router lets you expose stable logical model names to your applications, then route requests to one or more upstream model providers with priority-based fallback, quota control, billing, logging, and an admin UI.

⁠Highlights

• OpenAI-compatible endpoints: /v1/chat/completions, /v1/embeddings, /v1/models
• Anthropic-compatible endpoints: /anthropic/v1/messages, /anthropic/v1/models
• Logical model routing with provider priority and automatic fallback
• Per-key balance, daily spend limit, QPS limit, and allowed model control
• Accurate token and cost tracking, including cache and reasoning tokens
• Request logs, statistics dashboard, latency, cost, users, channels, and error classes
• Optional request/response content audit logging
• Payload overrides for upstream model sources
• Optional image-content stripping for non-multimodal fallback models
• Built-in admin panel
• SQLite local mode, or MySQL + Redis server mode

⁠Quick Start

⁠Local mode with SQLite

docker run -d \
  --name llm-router \
  -p 8000:8000 \
  -e APP_ENCRYPTION_KEY="your-fernet-key" \
  -e SESSION_SECRET="your-session-secret" \
  liuzhenghua/llm-router:latest

Open the admin panel:

http://localhost:8000/admin/login

⁠Server Mode

Use MySQL and Redis for multi-instance or production deployments.

docker run -d \
  --name llm-router \
  -p 8000:8000 \
  -e APP_ENCRYPTION_KEY="your-fernet-key" \
  -e SESSION_SECRET="your-session-secret" \
  -e MYSQL_URL="mysql://llm_router@mysql:3306/llm_router" \
  -e MYSQL_PASSWORD="your-mysql-password" \
  -e REDIS_URL="redis://redis:6379/0" \
  liuzhenghua/llm-router:latest

⁠Required Environment Variables

⁠SDK Usage

Use it as a drop-in OpenAI-compatible gateway:

from openai import OpenAI

client = OpenAI(
    base_url="http://your-host/v1",
    api_key="your-llm-router-key",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Or as an Anthropic-compatible gateway:

import anthropic

client = anthropic.Anthropic(
    base_url="http://your-host/anthropic",
    api_key="your-llm-router-key",
)

message = client.messages.create(
    model="claude-3-5-sonnet",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)

⁠Repository

Source code and documentation:

https://github.com/liuzhenghua/llm-router⁠

⁠License

AGPLv3