LLM/VLM inference with oobabooga textgen on runpod
491
/workspace)| Component | Version |
|---|---|
| OS | Ubuntu v24.04 x86_64 |
| Python | 3.13.3 |
| CUDA | 12.8.1 |
| Textgen | 4.9 |
| Code-Server | latest |
run-textgen is built from ls250824/python-cuda-ubuntu-develop:23052026, which is based on nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04.
The base image provides Ubuntu 24.04, CUDA 12.8.1, cuDNN development libraries, Python 3.13.3 compiled from source, and a virtual environment at /opt/venv. It includes build tools and common RunPod utilities. PyTorch is installed later through the textgen requirements, not by the base image.
| Token | Environment Variable | Example | Required |
|---|---|---|---|
| Hugging Face | HF_TOKEN | token | Optional |
| Code Server | PASSWORD | password | Optional |
| textgen | GRADIO_AUTH | user:password | Optional |
| Model Type | Hugging Face URL Variable | GGUF File Variable |
|---|---|---|
| GGUF Model | HF_MODEL_GGUF[1-6] | HF_MODEL_GGUF_FILE[1-6] |
| Model Type | Hugging Face URL Variable | GGUF File Variable |
|---|---|---|
| GGUF MMPROJ | HF_MMPROJ_GGUF[1-6] | HF_MMPROJ_GGUF_FILE[1-6] |
At startup, the script reads the total memory of every GPU with nvidia-smi and uses the largest value. A GPU with more than VRAM_THRESHOLD GiB selects HVRAM; all other GPUs select LVRAM. The default threshold is 36 GiB.
| Profile | GGUF Repository | GGUF File | MMPROJ Repository | MMPROJ File |
|---|---|---|---|---|
| HVRAM | HF_MODEL_HVRAM_GGUF[1-6] | HF_MODEL_HVRAM_GGUF_FILE[1-6] | HF_MODEL_HVRAM_MMPROJ_GGUF[1-6] | HF_MODEL_HVRAM_MMPROJ_GGUF_FILE[1-6] |
| LVRAM | HF_MODEL_LVRAM_GGUF[1-6] | HF_MODEL_LVRAM_GGUF_FILE[1-6] | HF_MODEL_LVRAM_MMPROJ_GGUF[1-6] | HF_MODEL_LVRAM_MMPROJ_GGUF_FILE[1-6] |
Only variables belonging to the selected profile are downloaded. The existing HF_MODEL_GGUF[1-6] and HF_MMPROJ_GGUF[1-6] variables remain VRAM-independent and are always processed. Set VRAM_THRESHOLD to change the boundary.
| Model Type | Hugging Face URL Variable | Destination Subfolder Variable | Include Filter Variable | Exclude Filter Variable |
|---|---|---|---|---|
| Transformers | HF_MODEL[1-6] | HF_MODEL_DIR[1-6] | HF_MODEL_INCLUDE[1-6] | HF_MODEL_EXCLUDE[1-6] |
HF_MODEL_INCLUDE[1-6] and HF_MODEL_EXCLUDE[1-6] are optional and map to hf download --include and hf download --exclude. If no destination subfolder is set, filtered files are downloaded into /workspace/textgen/user_data/models/.
All Hugging Face model downloads start with the Xet backend. Before downloading, the script performs an hf download --dry-run to report the expected total size. During the transfer, RunPod logs show downloaded gigabytes and transfer speed. If the download has no output or file growth for 300 seconds, the stalled process is stopped and retried automatically with Xet disabled (plain HTTP).
| Environment Variable | Description | Default |
|---|---|---|
HF_DOWNLOAD_STALL_TIMEOUT | Seconds without download activity before the current attempt is stopped | 300 |
HF_DOWNLOAD_KILL_AFTER | Grace period in seconds before a stalled process is force-killed | 30 |
| Model Type | Hugging Face URL Variable | Revision | Destination Subfolder Variable |
|---|---|---|---|
| EXL | HF_EXL[1-6] | HF_EXL_REVISION[1-6] | HF_EXL_DIR[1-6] |
textgen is checked locally on port 7860 before model provisioning continues. TEXTGEN_START_MAX_TRIES controls the maximum number of checks, with five seconds between attempts; the default is 60 (about five minutes).
| Service | Port |
|---|---|
| Code Server | 9000 (HTTP) |
| SSH/SCP | 22 (TCP) |
| Gradio | 7860 (HTTP) |
Content type
Image
Digest
sha256:4b1a0c996…
Size
12.6 GB
Last updated
about 1 month ago
docker pull ls250824/run-textgen:19082026