exclusivebinary

Kong OSS with custom OIDC plugin. Github: https://github.com/michaeltansg/kong-oidc-plugin

API service for providing embedding.

Pass the database connection string to through an environment variable "DATABASE_URL".

Supported platforms: linux/amd64, linux/arm64

Service for meta-llama/Llama-2-70b-chat-hf (SpQR [int-4] quantisation) using 1 worker (GPU).

Service for meta-llama/Llama-2-7b-chat-hf (8-bit quantisation) using 1 worker (GPU).

Service for meta-llama/Llama-2-7b-chat-hf (full-precision) distributed over 2 workers (GPUs).

Service for meta-llama/Llama-2-13b-chat-hf (8-bit quantisation) distributed over 2 workers (GPUs)

Service for meta-llama/Llama-2-7b-chat-hf (full-precision) using 1 worker (GPU).