Sign inSign up

robcowart/aiproxy

By robcowart

•Updated 4 months ago

Image
0

252

robcowart/aiproxy repository overview

⁠AIproxy

A model-aware, OpenAI-compatible reverse proxy for pools of local/remote LLM backends (llama.cpp, vLLM, OpenAI, Anthropic, Google, Ollama).

WARNING! This is still a work-in-progress. My main use-case is providing a single OpenAI API-compatible service from which I can reach multiple llama-server (llama.cpp) instances in my AI lab. It is working really well for this use-case so far. However I have not yet thoroughly tested the other backend LLM services. If you encounter any problems, please open an issue or PR.

⁠Supported backend schema values

SchemaDescription
llamacppllama.cpp servers (OpenAI-compatible wire format, /health probe). Also works for vLLM, which exposes an OpenAI-compatible server with the same /health probe.
openaiOpenAI API and OpenAI-compatible providers
anthropicAnthropic Messages API (/v1/messages)
googleGoogle Gemini API
ollamaNative Ollama API (/api/chat, /api/embed, /api/tags) with NDJSON streaming. Supports chat_completions and embeddings. rerank is unsupported. Ollama also exposes an OpenAI-compatible /v1/* shim, for which you can use the openai schema.

⁠Supported Endpoints

EndpointAPI path
chat_completions/v1/chat/completions
embeddings/v1/embeddings
rerank/v1/rerank
infill/infill

infill is a byte-for-byte pass-through to llama.cpp's native /infill⁠ endpoint and is only valid with schema: llamacpp. Both the request body and the response body (including SSE stream frames) are forwarded unchanged so the wire shape matches llama.cpp exactly. Unlike the other routes there is no /v1/ alias, no canonical OpenAI envelope, and no data: [DONE] sentinel on the stream.

⁠Example config.yaml

server:
  host: '0.0.0.0'
  port: 8080
  api_key: '012345678901234567890123456789'

  # TLS/HTTPS Configuration (optional)
  tls:
    enabled: false  # Set to true to enable HTTPS
    # cert_file: '/path/to/server-cert.pem'  # Server certificate file
    # key_file: '/path/to/server-key.pem'    # Server private key file

log:
  level: 'info'   # debug, info, warn, error
  format: 'json'  # json, console

pools:
  - model: 'qwen3.5-122b'
    endpoint: 'chat_completions' # chat_completions, embeddings, rerank, infill
    schema: 'llamacpp' # anthropic, google, openai, llamacpp, ollama
    instances:
      - url: 'http://192.0.2.11:8080'
        api_key: 'abcdefabcdefabcdefabcdef'
        # tls:
        #   ca_file: '/path/to/backend-ca.pem'
        #   insecure_skip_verify: false
      - url: 'http://192.0.2.12:8080'
        api_key: 'abcdefabcdefabcdefabcdef'
        tls:
          ca_file: '/path/to/backend-ca.pem'
          insecure_skip_verify: false
    session_timeout: 300       # seconds
    health_check_interval: 30  # seconds

  - model: 'nemotron-embed-1b-v2'
    endpoint: 'embeddings' # chat_completions, embeddings, rerank, infill
    schema: 'llamacpp' # anthropic, google, openai, llamacpp, ollama
    instances:
      - url: 'http://192.0.2.13:8080'
        api_key: 'fedcbafedcbafedcbafedcba'
    session_timeout: 300       # seconds
    health_check_interval: 30  # seconds

⁠Example docker-compose.yaml

services:
  aiproxy:
    image: robcowart/aiproxy:latest
    container_name: aiproxy
    restart: unless-stopped
    volumes:
      - ./config.yaml:/etc/aiproxy/config.yaml:ro
    ports:
      - 8080:8080/tcp

Tag summary

Content type

Image

Digest

sha256:3cd32a8b9…

Size

8.2 MB

Last updated

4 months ago

docker pull robcowart/aiproxy