cliftonzac/bedrock-access-gateway-lambda repository overview

⁠Bedrock Access Gateway

OpenAI-compatible RESTful APIs for Amazon Bedrock

⁠What's New 🔥

API Gateway Response Streaming Support - You can now deploy with Amazon API Gateway REST API instead of ALB, enabling true response streaming for better latency and cost optimization. See Deployment Options⁠ for details.

Latest Models Supported:

• Claude 4.5 Family: Opus 4.5, Sonnet 4.5, Haiku 4.5 - Anthropic's most intelligent models with enhanced coding and agent capabilities
• Amazon Nova: Nova Micro, Nova Lite, Nova Pro, Nova Premier - Amazon's native foundation models with multimodal support
• DeepSeek: DeepSeek-R1 (reasoning), DeepSeek-V3.1 - Advanced reasoning and general-purpose models
• Qwen 3: Qwen3-32B, Qwen3-235B, Qwen3-Coder-30B, Qwen3-Coder-480B - Alibaba's latest language and coding models
• OpenAI OSS: gpt-oss-20b, gpt-oss-120b - Open-source GPT models available via Bedrock

It also supports reasoning for Claude 4/4.5 (extended thinking and interleaved thinking) and DeepSeek R1. Check How to Use⁠ for more details. You need to first run the Models API to refresh the model list.

⁠Overview

Amazon Bedrock offers a wide range of foundation models (such as Claude 3 Opus/Sonnet/Haiku, Llama 2/3, Mistral/Mixtral, etc.) and a broad set of capabilities for you to build generative AI applications. Check the Amazon Bedrock⁠ landing page for additional information.

Sometimes, you might have applications developed using OpenAI APIs or SDKs, and you want to experiment with Amazon Bedrock without modifying your codebase. Or you may simply wish to evaluate the capabilities of these foundation models in tools like AutoGen etc. Well, this repository allows you to access Amazon Bedrock models seamlessly through OpenAI APIs and SDKs, enabling you to test these models without code changes.

If you find this GitHub repository useful, please consider giving it a free star ⭐ to show your appreciation and support for the project.

Features:

• Support streaming response via server-sent events (SSE)
• Support Model APIs
• Support Chat Completion APIs
• Support Tool Call
• Support Embedding API
• Support Multimodal API
• Support Cross-Region Inference
• Support Application Inference Profiles (new)
• Support Reasoning (new)
• Support Interleaved thinking (new)
• Support Prompt Caching (new)

Please check Usage Guide⁠ for more details about how to use the new APIs.

⁠Get Started

⁠Prerequisites

Please make sure you have met below prerequisites:

• Access to Amazon Bedrock foundation models.

⁠Architecture

The following diagram illustrates the reference architecture. It uses Amazon API Gateway response streaming⁠ with Lambda for SSE support.

⁠Deployment Options

You can also use Lambda Function URL as an alternative, see example⁠

⁠Pre-built Docker Hub images

Container images are built by the Publish Docker image to Docker Hub⁠ GitHub Actions workflow on every push to main and every v*.*.* tag. Two images are published:

Tags: latest (HEAD of main), semver (1.2.3, 1.2, 1) on release tags, and short commit SHA.

Run locally with the pre-built ECS image:

docker run --rm -p 8000:8080 \
  -e API_KEY=my-local-key \
  -e AWS_REGION=us-west-2 \
  -v ~/.aws:/home/appuser/.aws:ro \
  <your-dockerhub-username>/bedrock-access-gateway:latest

Base URL: http://localhost:8000/api/v1. Use any valid AWS credential source — an AWS_PROFILE, explicit AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars, or a mounted ~/.aws dir as shown above.

Enabling the workflow on a fork: add two repository secrets — DOCKERHUB_USERNAME (your Docker Hub namespace) and DOCKERHUB_TOKEN (a Docker Hub personal access token with Read, Write, Delete scope). Trigger with a push, a v*.*.* tag, or manually via Actions → Publish Docker image to Docker Hub → Run workflow.

Staying in sync with upstream: a companion Sync from upstream⁠ workflow runs daily (06:00 UTC) and on manual dispatch. It merges aws-samples/bedrock-access-gateway:main into this fork's main (failing loudly on conflict so you can resolve by hand) and pushes the result. A successful sync then chains into the Docker Hub publish workflow via workflow_run, so new upstream commits become new image tags automatically.

⁠Deployment

Please follow the steps below to deploy the Bedrock Proxy APIs into your AWS account. Only supports regions where Amazon Bedrock is available (such as us-west-2). The deployment will take approximately 10-15 minutes 🕒.

Step 1: Create your own API key in Secrets Manager (MUST)

1. Open the AWS Management Console and navigate to the AWS Secrets Manager service.

2. Click on "Store a new secret" button.

3. In the "Choose secret type" page, select:

   Secret type: Other type of secret Key/value pairs:

   • Key: api_key
   • Value: Enter your API key value

   Click "Next"

4. In the "Configure secret" page: Secret name: Enter a name (e.g., "BedrockProxyAPIKey") Description: (Optional) Add a description of your secret

5. Click "Next" and review all your settings and click "Store"

After creation, you'll see your secret in the Secrets Manager console. Make note of the secret ARN.

Step 2: Build and push container images to ECR

1. Clone this repository:

   git clone https://github.com/aws-samples/bedrock-access-gateway.git
   cd bedrock-access-gateway
   

   Copy

2. Run the build and push script:

   cd scripts
   bash ./push-to-ecr.sh
   

   Copy

3. Follow the prompts to configure:

   • ECR repository names (or use defaults)
   • Image tag (or use default: latest)
   • AWS region (or use default: us-east-1)

4. The script will build and push both Lambda and ECS/Fargate images to your ECR repositories.

5. Important: Copy the image URIs displayed at the end of the script output. You'll need these in the next step.

Step 3: Deploy the CloudFormation stack

1. Download the CloudFormation template you want to use:

   • For API Gateway + Lambda: deployment/BedrockProxy.template⁠
   • For ALB + Fargate: deployment/BedrockProxyFargate.template⁠

2. Sign in to AWS Management Console and navigate to the CloudFormation service in your target region.

3. Click "Create stack" → "With new resources (standard)".

4. Upload the template file you downloaded.

5. On the "Specify stack details" page, provide the following information:

   • Stack name: Enter a stack name (e.g., "BedrockProxyAPI")
   • ApiKeySecretArn: Enter the secret ARN from Step 1
   • ContainerImageUri: Enter the ECR image URI from Step 2 output
   • DefaultModelId: (Optional) Change the default model if needed

   Click "Next".

6. On the "Configure stack options" page, you can leave the default settings or customize them according to your needs. Click "Next".

7. On the "Review" page, review all details. Check the "I acknowledge that AWS CloudFormation might create IAM resources" checkbox at the bottom. Click "Submit".

That is it! 🎉 Once deployed, click the CloudFormation stack and go to Outputs tab, you can find the API Base URL from APIBaseUrl, the value should look like http://xxxx.xxx.elb.amazonaws.com/api/v1.

⁠Troubleshooting

If you encounter any issues, please check the Troubleshooting Guide⁠ for more details.

⁠SDK/API Usage

All you need is the API Key and the API Base URL. If you didn't set up your own key following Step 1, the application will fail to start with an error message indicating that the API Key is not configured.

Now, you can try out the proxy APIs. Let's say you want to test Claude 3 Sonnet model (model ID: anthropic.claude-3-sonnet-20240229-v1:0)...

Example API Usage

export OPENAI_API_KEY=<API key>
export OPENAI_BASE_URL=<API base url>
# For older versions
# https://github.com/openai/openai-python/issues/624
export OPENAI_API_BASE=<API base url>

curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "anthropic.claude-3-sonnet-20240229-v1:0",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

Example SDK Usage

from openai import OpenAI

client = OpenAI()
completion = client.chat.completions.create(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(completion.choices[0].message.content)

Please check Usage Guide⁠ for more details about how to use embedding API, multimodal API and tool call.

⁠Application Inference Profiles

This proxy now supports Application Inference Profiles, which allow you to track usage and costs for your model invocations. You can use application inference profiles created in your AWS account for cost tracking and monitoring purposes.

Using Application Inference Profiles:

# Use an application inference profile ARN as the model ID
curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/your-profile-id",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

SDK Usage with Application Inference Profiles:

from openai import OpenAI

client = OpenAI()
completion = client.chat.completions.create(
    model="arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/your-profile-id",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(completion.choices[0].message.content)

Benefits of Application Inference Profiles:

• Cost Tracking: Track usage and costs for specific applications or use cases
• Usage Monitoring: Monitor model invocation metrics through CloudWatch
• Tag-based Cost Allocation: Use AWS cost allocation tags for detailed billing analysis

For more information about creating and managing application inference profiles, see the Amazon Bedrock User Guide⁠.

⁠Prompt Caching

This proxy now supports Prompt Caching for Claude and Nova models, which can reduce costs by up to 90% and latency by up to 85% for workloads with repeated prompts.

Supported Models:

• Claude models (Claude 3.5 Haiku, Claude 4, Claude 4.5, etc.)
• Nova models (Nova Micro, Nova Lite, Nova Pro, Nova Premier)

Enabling Prompt Caching:

You can enable prompt caching in two ways:

1. Globally via Environment Variable (set in ECS Task Definition or Lambda):

ENABLE_PROMPT_CACHING=true

1. Per-request via extra_body :

Python SDK:

from openai import OpenAI

client = OpenAI()

# Cache system prompts
response = client.chat.completions.create(
    model="global.anthropic.claude-haiku-4-5-20251001-v1:0",
    messages=[
        {"role": "system", "content": "You are an expert assistant with knowledge of..."},
        {"role": "user", "content": "Help me with this task"}
    ],
    extra_body={
        "prompt_caching": {"system": True}
    }
)

# Check cache hit
if response.usage.prompt_tokens_details:
    cached_tokens = response.usage.prompt_tokens_details.cached_tokens
    print(f"Cached tokens: {cached_tokens}")

cURL:

curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "global.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [
      {"role": "system", "content": "Long system prompt..."},
      {"role": "user", "content": "Question"}
    ],
    "extra_body": {
      "prompt_caching": {"system": true}
    }
  }'

Cache Options:

• "prompt_caching": {"system": true} - Cache system prompts
• "prompt_caching": {"messages": true} - Cache user messages
• "prompt_caching": {"system": true, "messages": true} - Cache both

Requirements:

• Prompt must be ≥1,024 tokens to enable caching
• Cache TTL is 5 minutes (resets on each cache hit)
• Nova models have a 20,000 token caching limit

For more information, see the Amazon Bedrock Prompt Caching Guide⁠.

⁠Other Examples

⁠LangChain

Make sure you use ChatOpenAI(...) instead of OpenAI(...)

# pip install langchain-openai
import os

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

chat = ChatOpenAI(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    temperature=0,
    openai_api_key=os.environ['OPENAI_API_KEY'],
    openai_api_base=os.environ['OPENAI_BASE_URL'],
)

template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate.from_template(template)
llm_chain = LLMChain(prompt=prompt, llm=chat)

question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"
response = llm_chain.invoke(question)
print(response)

⁠FAQs

⁠About Privacy

This application does not collect any of your data. Furthermore, it does not log any requests or responses by default.

⁠Why choose API Gateway vs ALB?

API Gateway + Lambda uses API Gateway response streaming⁠ with Lambda Web Adapter⁠ to support SSE streaming without requiring a VPC. This is a cost-effective, serverless option with up to 10 minutes timeout.

ALB + Fargate provides the lowest streaming latency with no cold starts, ideal for high-throughput workloads.

⁠Which regions are supported?

Generally speaking, all regions that Amazon Bedrock supports will also be supported, if not, please raise an issue in Github.

Note that not all models are available in those regions.

⁠Which models are supported?

You can use the Models API⁠ to get/refresh a list of supported models in the current region.

⁠Can I run this locally

Yes. Three options:

1. From the pre-built Docker Hub image — see Pre-built Docker Hub images⁠ above.
2. From source with Docker Compose:

   OPENAI_API_KEY=my-local-key docker-compose up --build
   

   Copy
3. From source with uvicorn directly, under the src folder:

   API_KEY=my-local-key AWS_REGION=us-west-2 uvicorn api.app:app --host 0.0.0.0 --port 8000
   

   Copy

The API base url should look like http://localhost:8000/api/v1 in all cases.

⁠Any performance sacrifice or latency increase by using the proxy APIs

Compared with direct AWS SDK calls, the proxy architecture will add some latency. The default API Gateway + Lambda deployment provides good streaming performance with Lambda response streaming.

For lowest latency on streaming responses, consider the ALB + Fargate deployment option which eliminates cold starts and provides consistent performance.

⁠Any plan to support SageMaker models?

Currently, there is no plan to support SageMaker models. This may change provided there's a demand from customers.

⁠Any plan to support Bedrock custom models?

Fine-tuned models and models with Provisioned Throughput are currently not supported. You can clone the repo and make the customization if needed.

⁠How to upgrade?

To use the latest features, you need follow the deployment guide to redeploy the application. You can upgrade the existing CloudFormation stack to get the latest changes.

⁠Security

See CONTRIBUTING⁠ for more information.

⁠License

This library is licensed under the MIT-0 License. See the LICENSE file.