sodlinken/jairouter repository overview

⁠JAiRouter

Production-Ready AI Model Gateway

OpenAI-compatible API for unified routing, load balancing & failover
across Ollama, vLLM, GPUStack, Xinference, and more

⁠ ⁠ ⁠ ⁠

中文⁠ • Docs⁠ • English Docs⁠ • Discussions⁠

⁠Quick Start

# Start with Docker (no configuration needed)
docker run -d --name jairouter -p 8080:8080 sodlinken/jairouter:latest

# Open Web Console: http://localhost:8080/admin
# Default: admin / ChangeMeOnFirstStartup123456

⁠What is JAiRouter?

JAiRouter is a production-ready AI model gateway that provides a unified, OpenAI-compatible API for managing multiple LLM backends. It handles load balancing, rate limiting, circuit breaking, and failover — so you can focus on building applications, not managing infrastructure.

⁠Key Benefits

⁠Core Features

• 🔌 OpenAI-Compatible API — Drop-in replacement for OpenAI SDK, LangChain, LlamaIndex
• ⚖️ Smart Load Balancing — Round-robin, weighted, least-connections, IP-hash, consistent-hash
• 🛡️ Rate Limiting — Token bucket, leaky bucket, sliding window algorithms
• 🔥 Circuit Breaker — Auto failover with configurable thresholds and recovery
• 🔐 Authentication — JWT + API Key dual authentication with audit logging
• 📊 Observability — Prometheus metrics, OpenTelemetry tracing, real-time dashboards
• 💾 Persistence — Redis / H2 / File storage for distributed deployment
• 🎛️ Web Console — Visual management, version control, configuration rollback

⁠Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Your Application                         │
│                 (OpenAI SDK / LangChain / LlamaIndex)           │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
                                  ▼ OpenAI-Compatible API
┌─────────────────────────────────────────────────────────────────┐
│                         JAiRouter Gateway                        │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐   │
│  │ Routing │ │ Balance │ │  Limit  │ │ Circuit │ │   Auth  │   │
│  └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              Observability & Persistence                 │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────┬───────────────────────────────┘
                                  │
       ┌──────────────┬───────────┼───────────┬──────────────┐
       ▼              ▼           ▼           ▼              ▼
   ┌───────┐     ┌───────┐   ┌───────┐   ┌──────────┐   ┌───────┐
   │Ollama │     │ vLLM  │   │GPUStack│  │Xinference│   │ OpenAI │
   └───────┘     └───────┘   └───────┘   └──────────┘   └───────┘

⁠Supported AI Backends

⁠Usage Example

⁠Python with OpenAI SDK

from openai import OpenAI

# Point to JAiRouter instead of OpenAI
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"  # JAiRouter handles authentication
)

# Use any model from your configured backends
response = client.chat.completions.create(
    model="llama3.2",  # Routed to Ollama, vLLM, or GPUStack
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

⁠Add a Model Backend

# Via API - Add Ollama instance
curl -X POST http://localhost:8080/api/config/instance/add/chat \
  -H "Content-Type: application/json" \
  -H "Jairouter_token: your-jwt-token" \
  -d '{
    "name": "llama3.2",
    "baseUrl": "http://localhost:11434",
    "path": "/v1/chat/completions",
    "weight": 1
  }'

⁠Why Choose JAiRouter?

⁠vs Nginx

Nginx is a general-purpose web server. JAiRouter is purpose-built for AI/LLM workloads with OpenAI-compatible routing, circuit breaking, and model-aware load balancing.

⁠vs One-API

One-API focuses on API key management and billing. JAiRouter focuses on local model gateway with advanced resilience patterns and observability.

⁠vs LangChain

LangChain is an application framework. JAiRouter is an infrastructure layer that works beneath LangChain to provide routing, failover, and monitoring.

⁠Benchmarks

Performance overhead compared to direct backend access:

⁠Documentation

⁠Roadmap

• Core gateway functionality
• Multiple backend adapters (Ollama, vLLM, GPUStack, Xinference, LocalAI)
• Load balancing with multiple strategies
• Rate limiting algorithms
• Circuit breaker with auto recovery
• Web management console
• JWT + API Key authentication
• OpenTelemetry distributed tracing
• Configuration version control
• Plugin system for custom adapters
• GraphQL API support

⁠Contributing

We welcome contributions! See Contributing Guide⁠.

git clone https://github.com/Lincoln-cn/JAiRouter.git
cd JAiRouter/modelrouter
mvn clean package -DskipTests
java -jar target/modelrouter.jar

⁠Support

• Documentation: https://jairouter.com⁠
• Issues: GitHub Issues⁠
• Discussions: GitHub Discussions⁠

⁠License

JAiRouter is released under the Apache 2.0 License⁠.

Star ⭐ this repo if you find it useful!

Made with ❤️ by the JAiRouter Team