ivangabriele/llm
OpenAI API-Compatible ready-to-deploy LLM Server.
Image
0
577
ivangabriele/llm repository overview
Docker Image ―
OpenAI API-Compatible Pre-loaded LLM Server
Docker images are based on Nvidia CUDA images. LLMs are pre-loaded and served via vLLM.
Environment Variables
TENSOR_PARALLEL_SIZE: Number of GPUs to use. Default:1.
Port
The OpenAI API is exposed on port 8000.
Tags & Deployment Links
Note
The VRAM column is the minimum required amount of VRAM used by the model on a single GPU.
| Tag | Model | RunPod | Vast.ai | VRAM |
|---|---|---|---|---|
ivangabriele/llm:lmsys__vicuna-13b-v1.5-16k |  |  |  | 26GB |
ivangabriele/llm:open-orca__llongorca-13b-16k | | | | 26GB |
Roadmap
- Add more popular models.
- Start the server in background to allow for SSH access.
Tag summary
Content type
Image
Digest
sha256:0da890f0d…
Size
37.2 GB
Last updated
almost 3 years ago
docker pull ivangabriele/llm:lmsys__vicuna-13b-v1.5-16k