portkeyai/gateway repository overview

⁠AI Gateway

⁠Route to 100+ LLMs with 1 fast & friendly API.

Portkey's AI Gateway⁠ is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.

✅  Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
✅  Load balance across multiple models, providers, and keys
✅  Fallbacks make sure your app stays resilient
✅  Automatic Retries with exponential fallbacks come by default
✅  Plug-in middleware as needed
✅  Battle tested over 100B tokens

⁠Deploy using Docker

Run using Docker directly:

docker run -d -p 8787:8787 portkeyai/gateway:latest

For more information on the Docker image, check here⁠

⁠Deploy using Docker Compose

1. Download compose file from the repo:

wget "https://raw.githubusercontent.com/Portkey-AI/gateway/main/docker-compose.yaml"

1. Run:

docker compose up -d

1. The service is now running and listening on port 8787

⁠Usage

Let's try making a chat completions call to OpenAI through the AI gateway:

curl '127.0.0.1:8787/v1/chat/completions' \
  -H 'x-portkey-provider: openai' \
  -H "Authorization: Bearer $OPENAI_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "gpt-4"}'

Full list of supported SDKs⁠

⁠Supported Providers

	Provider	Support	Stream	Supported Endpoints
	OpenAI	✅	✅	/completions, /chat/completions,/embeddings, /assistants, /threads, /runs, /images/generations, /audio/*
	Azure OpenAI	✅	✅	/completions, /chat/completions,/embeddings
	Anyscale	✅	✅	/chat/completions
	Google Gemini & Palm	✅	✅	/generateMessage, /generateText, /embedText
	Anthropic	✅	✅	/messages, /complete
	Cohere	✅	✅	/generate, /embed, /rerank
	Together AI	✅	✅	/chat/completions, /completions, /inference
	Perplexity	✅	✅	/chat/completions
	Mistral	✅	✅	/chat/completions, /embeddings
	Nomic	✅	✅	/embeddings
	AI21	✅	✅	/complete, /chat, /embed
	Stability AI	✅	✅	/generation/{engine_id}/text-to-image
	DeepInfra	✅	✅	/inference
	Ollama	✅	✅	/chat/completions

⁠Features

⁠Unified API Signature⁠ Connect with 100+ LLM using OpenAI's API signature. The AI gateway handles the request, response and error transformations so you don't have to make any changes to your code. You can use the OpenAI SDK itself to connect to any of the supported LLMs.

⁠Fallback⁠ Don't let failures stop you. The Fallback feature allows you to specify a list of Language Model APIs (LLMs) in a prioritized order. If the primary LLM fails to respond or encounters an error, Portkey will automatically fallback to the next LLM in the list, ensuring your application's robustness and reliability.

⁠Automatic Retries⁠ Temporary issues shouldn't mean manual re-runs. AI Gateway can automatically retry failed requests upto 5 times. We apply an exponential backoff strategy, which spaces out retry attempts to prevent network overload.

⁠Load Balancing⁠ Distribute load effectively across multiple API keys or providers based on custom weights. This ensures high availability and optimal performance of your generative AI apps, preventing any single LLM from becoming a performance bottleneck.

⁠Supported SDKs

⁠Community

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog⁠
• Chat live with us on Discord⁠
• Follow us on Twitter⁠
• Connect with us on LinkedIn⁠