mohammedtv/llmproxy repository overview

⁠LLMProxy

Self-hosted, OpenAI-compatible LLM gateway built with C# and .NET 10. Applications use one base URL and gateway API keys; LLMProxy manages upstream credentials, model routing, quotas and usage accounting. MIT licensed.

Source and full README⁠ · Releases and upgrades⁠ · Configuration⁠ · API reference⁠ · Operations⁠

⁠Images

Version 0.2.0 is built from commit 1911fc8⁠. The image passed the repository's CI checks⁠, including native AMD64 and ARM64 container and SDK integration tests.

docker pull mohammedtv/llmproxy:0.2.0

Version 0.2.0's multi-platform digest is sha256:a8bd1b3e93b12549733b86fbb0be9809704924e2a2e27bee075df4c7ab56d3db. GitHub source tags have a v prefix, while Docker Hub version tags omit it. See the release guide⁠ for tag conventions and verified builds.

⁠Quick start

Create a local .env file containing at least one upstream credential:

OPENAI_API_KEY=your-provider-key

Run the gateway with persistent SQLite storage:

docker run -d \
  --name llmproxy \
  --restart unless-stopped \
  -p 4000:4000 \
  --env-file .env \
  -v llmproxy-data:/data \
  mohammedtv/llmproxy:0.2.0

docker exec llmproxy dotnet LLMProxy.Server.dll keys create \
  --owner my-app --models fast,reasoning,embeddings --rpm 60

The CLI returns a JSON object containing a one-time gateway key beginning with llmp_sk_. Store that key for your application. Upstream provider keys stay on the gateway.

export LLMPROXY_API_KEY='the-key-returned-by-keys-create'
curl http://localhost:4000/v1/models \
  -H "Authorization: Bearer $LLMPROXY_API_KEY"

Use http://localhost:4000/v1 as the OpenAI SDK base URL and your gateway key as the API key. Model aliases and capabilities depend on configured providers; unavailable aliases are omitted from /v1/models. See the Python, TypeScript and C# examples⁠.

⁠Multiple keys per provider

Every API-key adapter supports up to 64 distinct keys per provider account. For example, put this in the .env passed to docker run:

LLMProxy__Providers__OpenAI__ApiKeys__0=provider-key-1
LLMProxy__Providers__OpenAI__ApiKeys__1=provider-key-2
LLMProxy__Providers__OpenAI__ApiKeyCooldownSeconds=30

A nonempty pool replaces the single OPENAI_API_KEY. Requests rotate among available keys. Authentication failures and rate limits switch to another key and cool the affected credential; transient failures use bounded retries before failover. Each key has a separate circuit breaker. Rotation and cooldowns are local to each gateway process.

Named provider accounts support separate endpoints, credentials, prices and concurrency scopes for the same adapter. Compose requires these indexed variables to be explicitly forwarded in a service environment override. See pool configuration and Compose examples⁠.

⁠Features

Provider support differs by operation and model. Consult the compatibility matrix⁠ and API boundaries⁠. Responses is stateless and supported through OpenAI/Foundry. Files are for gateway batches. Image generation, speech generation/transcription and legacy Completions are not exposed.

⁠Docker Compose with PostgreSQL and Redis

Obtain the repository and copy .env.example to .env. Set provider credentials, separate POSTGRES_PASSWORD and REDIS_PASSWORD values, and this image setting:

LLMPROXY_IMAGE=mohammedtv/llmproxy:0.2.0

From the checkout:

docker compose pull
docker compose up -d --no-build
docker compose exec llmproxy dotnet LLMProxy.Server.dll keys create \
  --owner my-app --models fast,reasoning,embeddings --rpm 60 \
  --monthly-token-limit 1000000 --monthly-budget 20
curl http://localhost:4000/health/ready

Compose runs PostgreSQL 16 and Redis 7 with persistent volumes and publishes only the gateway port. PostgreSQL plus Redis is required for multiple gateway replicas. The standalone SQLite volume is intended for one instance.

⁠Container configuration

Set LLMPROXY_ADMIN_KEY before starting the container, then open /admin to create the first local administrator. Local operator accounts continue to work if the shared bootstrap secret is later removed.

/health/live checks the process; /health/ready checks storage, Redis, provider configuration and shutdown state. Health checks do not call a live LLM API. The image includes a Docker health check. CLI commands include keys create, migrate, reservations recover and healthcheck.

For production, pin a version or digest, persist and back up the database, configure TLS at a trusted reverse proxy, and use the operations guide⁠ for SSE proxy settings, migrations, telemetry and recovery.