[{"_1":2,"_162":-5,"_163":-5},"loaderData",{"_3":4,"_18":19,"_25":26,"_141":142},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16,"_17":11},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","dockerRebrandTour","routes/_layout",{"_20":-5,"_21":22,"_23":11,"_24":22},"username","isAskGordonEnabled",true,"showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_129":-5,"_130":11,"_131":22,"_132":11,"_133":11,"_134":-5,"_135":11,"_136":-5,"_137":138,"_139":91,"_140":-7},"canonicalUrl","https://hub.docker.com/r/weyp/llama-cpp-python-server-cuda/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","weyp","reponame","llama-cpp-python-server-cuda","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":11,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2024-04-09T10:58:26.407742Z","description","Ready to use llama.cpp python server with OpenAI compatible API and autodownloader for HF models","is_private","last_updated","2024-07-01T17:05:48.686571Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",1590,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# llama-cpp-python\n\n**Docker container for llama-cpp-python - a python binding for [llama.cpp](https://github.com/ggerganov/llama.cpp).**\n\n## Index\n\n1. [Usage](#usage) \n 1.1 [docker run](#docker-run) \n 1.2 [docker-compose.yml](#docker-compose)\n2. [Environment Variables](#environment-variables)\n3. [Volumes](#volumes)\n4. [Ports](#ports)\n5. [API Endpoints](#endpoints)\n6. [Find Me](#findme)\n7. [License](#license)\n\n## 1 Usage \n\n**IMPORTANT: you need to add SYS_RESOURCE capability to enable MLOCK support**\n\n```shell\n# for docker run:\ndocker run -d --cap-add SYS_RESOURCE ...\n\n# for docker compose:\nversion: '3.9'\n llama-cpp-python:\n image: weyp/llama-cpp-python-server-cuda:latest\n container_name: llama-cpp-python\n cap_add:\n - SYS_RESOURCE\n```\n\n### 1.1 docker run \n\n**Example 1 - run model from huggingface:** \n**This is the recommended way to use this container !!!**\n\n```shell\ndocker run -d \\\n --name llama-cpp-python \\\n --cap-add SYS_RESOURCE \\\n -e MODEL_DOWNLOAD=\"True\" \\\n -e MODEL_REPO=\"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\" \\\n -e MODEL=\"mistral-7b-instruct-v0.2.Q4_K_M.gguf\" \\\n -e MODEL_ALIAS=\"mistral-7b-instruct\" \\\n -e CHAT_FORMAT=\"mistral\" \\\n -p 8000:8000/tcp \\\n weyp/llama-cpp-python-server-cuda:latest\n```\n\n**Example 2 - run own model from local file:**\n\n```shell\ndocker run -d \\\n --name llama-cpp-python \\\n --cap-add SYS_RESOURCE \\\n -e MODEL_DOWNLOAD=\"False\" \\\n -e MODEL_REPO=\"local\" \\\n -e MODEL=\"mistral-7b-instruct-v0.2.Q4_K_M.gguf\" \\\n -e MODEL_ALIAS=\"mistral-7b-instruct\" \\\n -e CHAT_FORMAT=\"mistral\" \\\n -v /path/to/your/model/mistral-7b-instruct-v0.2.Q4_K_M.gguf:/model/mistral-7b-instruct-v0.2.Q4_K_M.gguf \\\n -p 8000:8000/tcp \\\n weyp/llama-cpp-python-server-cuda:latest\n```\n\n**Example 3 - run with arguments (most environment variables will be ignored):** \n**arguments will be executed like this:** \n**/venv/bin/python3 -B -m llama_cpp.server --host 0.0.0.0 \\**\n\n```shell\ndocker run -d \\\n --name llama-cpp-python \\\n --cap-add SYS_RESOURCE \\\n -e MODEL_DOWNLOAD=\"False\" \\\n -v /path/to/your/model/mistral-7b-instruct-v0.2.Q4_K_M.gguf:/model/mistral-7b-instruct-v0.2.Q4_K_M.gguf \\\n -p 8000:8000/tcp \\\n weyp/llama-cpp-python-server-cuda:latest \\\n --model /model/mistral-7b-instruct-v0.2.Q4_K_M.gguf \\\n --n_ctx 1024 \\\n ...\n```\n\n**Example 4 - show help:**\n\n```shell\ndocker run --rm \\\n --name llama-cpp-python \\\n weyp/llama-cpp-python-server-cuda:latest \\\n --help\n```\n\n### 1.2 docker-compose.yml \n\n```yaml\nversion: '3.9'\n\nservices:\n llama-cpp-python:\n image: weyp/llama-cpp-python-server-cuda:latest\n container_name: llama-cpp-python\n cap_add:\n - SYS_RESOURCE\n environment:\n MODEL_DOWNLOAD: \"True\"\n MODEL_REPO: \"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\"\n MODEL: \"mistral-7b-instruct-v0.2.Q4_K_M.gguf\"\n MODEL_ALIAS: \"mistral-7b-instruct\"\n CHAT_FORMAT: \"mistral\"\n ports:\n - 8000:8000/tcp\n```\n\n### 2 Environment Variables \n\n- `TZ` - Specifies the server timezone - **default: UTC**\n- `MODEL_DOWNLOAD` - If True, downloads MODEL file from Huggingface MODEL_REPO- **default: true**\n- `MODEL_REPO` - The huggingface repo name. Set to `local` if MODEL was mounted locally - **default: TheBloke/Llama-2-7B-Chat-GGUF**\n- `MODEL` - **MANDATORY**: The model filename - **default: llama-2-7b-chat.Q4_K_M.gguf**\n- `MODEL_ALIAS` - The alias of the model to use for generating completions - **default: llama-2-7b-chat**\n- `SEED` - Random seed. -1 for random - **default: 4294967295**\n- `N_CTX` - The context size - **default: 2048**\n- `N_BATCH` - The batch size to use per eval - **default: 512**\n- `N_GPU_LAYERS` - The number of layers to put on the GPU. The rest will be on the CPU - **default: 0**\n- `MAIN_GPU` - Main GPU to use - **default: 0**\n- `TENSOR_SPLIT` - Split layers across multiple GPUs in proportion.\n- `ROPE_FREQ_BASE` - RoPE base frequency - **default: 0.0**\n- `ROPE_FREQ_SCALE` - RoPE frequency scaling factor - **default: 0.0**\n- `MUL_MAT_Q` - if true, use experimental mul_mat_q kernels - **default: True**\n- `LOGITS_ALL` - Whether to return logits - **default: True**\n- `VOCAB_ONLY` - Whether to only return the vocabulary - **default: False**\n- `USE_MMAP` - Use mmap - **default: True**\n- `USE_MLOCK` - Use mlock - **default: True**\n- `EMBEDDING` - Whether to use embeddings - **default: True**\n- `N_THREADS` - The number of threads to use - **default: 4**\n- `LAST_N_TOKENS_SIZE` - Last n tokens to keep for repeat penalty calculation - **default: 64**\n- `LORA_BASE` - Optional path to base model, useful if using a quantized base model and you want to apply LoRA to an f16 model.\n- `LORA_PATH` - Path to a LoRA file to apply to the model.\n- `NUMA` - Enable NUMA support - **default: False**\n- `CHAT_FORMAT` - Chat format to use - **default: llama-2**\n- `CACHE` - Use a cache to reduce processing times for evaluated prompts - **default: False**\n- `CACHE_TYPE` - The type of cache to use. Only used if cache is True - **default: ram**\n- `CACHE_SIZE` - The size of the cache in bytes. Only used if cache is True - **default: 2147483648**\n- `VERBOSE` - Whether to print debug information - **default: True**\n- `HOST` - Listen address - **default: 0.0.0.0**\n- `PORT` - Listen port - **default: 8000**\n- `INTERRUPT_REQUESTS` - Whether to interrupt requests when a new request is received - **default: True**\n- `HF_TOKEN` - Huggingface Token for private repos - **default: None**\n\n### 3 Volumes \n\n- `/model` - model directory -> **map your llama (\\*.gguf) models here**\n\n### 4 Ports \n\n- `8000/tcp` - API Port\n\n### 5 API Endpoints \n\n**visit [abetlen's documentation](https://abetlen.github.io/llama-cpp-python/) or** \n**see [http://llama-cpp-python:8000/docs](http://localhost:8000/docs) for more information**\n\n- `/v1/engines/copilot-codex/completions` - POST - Create Completion\n\n```json\n{\n \"prompt\": \"\\n\\n### Instructions:\\nWhat is the capital of France?\\n\\n### Response:\\n\",\n \"stop\": [\"\\n\", \"###\"]\n}\n```\n\n- `/v1/completions` - POST - Create Completion\n\n```json\n{\n \"prompt\": \"\\n\\n### Instructions:\\nWhat is the capital of France?\\n\\n### Response:\\n\",\n \"stop\": [\"\\n\", \"###\"]\n}\n```\n\n- `/v1/embeddings` - POST - Create Embedding\n\n```json\n{\n \"input\": \"The food was delicious and the waiter...\"\n}\n```\n\n- `/v1/chat/completions` - POST - Create Chat Completion\n\n```json\n{\n \"messages\": [\n {\n \"content\": \"You are a helpful assistant.\",\n \"role\": \"system\"\n },\n {\n \"content\": \"What is the capital of France?\",\n \"role\": \"user\"\n }\n ]\n}\n```\n\n- `/v1/models` - GET - Get Models\n **response:**\n\n```json\n{\n \"object\": \"list\",\n \"data\": [\n {\n \"id\": \"llama-2-7b-chat\",\n \"object\": \"model\",\n \"owned_by\": \"me\",\n \"permissions\": []\n }\n ]\n}\n```\n\n\n### 7 License \n\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) - This project is licensed under the GNU General Public License - see the [gpl-3.0](https://www.gnu.org/licenses/gpl-3.0.en.html) for details.","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":22,"_83":11,"_84":11},"read","write","admin","user","storage_size",8123879272,"orgData","recentTags",[91],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_101":102,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":22},"content_type","creator",26117428,"digest","sha256:f2bbbce095ebb0b753bb21332b43cabd04f1731b50364d13bee32b46e929c253","full_size",7241783972,"id",636367183,"images",[103],{"_104":105,"_95":96,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":98,"_63":66,"_117":-5},"architecture","amd64","features","","last_pulled","2026-10-29T03:59:45.01176447Z","last_pushed","2024-07-01T17:05:48Z","os","linux","os_features","os_version","size","variant","2024-07-01T17:05:47.630055Z","last_updater","last_updater_username","media_type","latest","repository",23783048,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxRunInCloud","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":143,"_150":151,"_137":160,"_161":11},{"_40":144,"_42":145,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":146,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":147,"_79":11,"_80":149,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":148},[78],{"_82":22,"_83":11,"_84":11},"tags",{"_152":64,"_153":-5,"_154":-5,"_155":156},"count","next","previous","results",[157],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_101":158,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":22},[159],{"_104":105,"_95":96,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":98,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

weyp/llama-cpp-python-server-cuda

Tags for weyp/llama-cpp-python-server-cuda