Sign inSign up

ls250824/run-textgen

By ls250824

Updated about 1 month ago

LLM/VLM inference with oobabooga textgen on runpod

Image
Machine learning & AI
0

491

ls250824/run-textgen repository overview

Run textgen on RunPod

This pod downloads models as specified in the environment variables

  • Models are automatically downloaded based on the specified paths in the environment configuration.
  • textgen starts first without automatically loading a model; provisioning runs afterwards, and downloaded models can then be loaded from the textgen interface.
  • Authentication credentials can be set via secrets for:
    • Code server authentication (not possible to switch off).
    • Hugging Face tokens for model access.
    • textgen gradio authentication.

Hardware Requirements

  • Recommended GPUs: RTX 4090, L40S , RTX A4500, RTX A5000, A40
  • Storage:
    • Volume: 80GB (/workspace)
    • Pod Volume: 5GB

Image setup

ComponentVersion
OSUbuntu v24.04 x86_64
Python3.13.3
CUDA12.8.1
Textgen4.9
Code-Serverlatest

Base image

run-textgen is built from ls250824/python-cuda-ubuntu-develop:23052026, which is based on nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04.

The base image provides Ubuntu 24.04, CUDA 12.8.1, cuDNN development libraries, Python 3.13.3 compiled from source, and a virtual environment at /opt/venv. It includes build tools and common RunPod utilities. PyTorch is installed later through the textgen requirements, not by the base image.

Environment Variables

Authentication Tokens
TokenEnvironment VariableExampleRequired
Hugging FaceHF_TOKENtokenOptional
Code ServerPASSWORDpasswordOptional
textgenGRADIO_AUTHuser:passwordOptional

📦 GGUF Model Downloads

Model TypeHugging Face URL VariableGGUF File Variable
GGUF ModelHF_MODEL_GGUF[1-6]HF_MODEL_GGUF_FILE[1-6]

📦 MMPROJ Downloads (multi modality)

Model TypeHugging Face URL VariableGGUF File Variable
GGUF MMPROJHF_MMPROJ_GGUF[1-6]HF_MMPROJ_GGUF_FILE[1-6]

VRAM-dependent GGUF downloads

At startup, the script reads the total memory of every GPU with nvidia-smi and uses the largest value. A GPU with more than VRAM_THRESHOLD GiB selects HVRAM; all other GPUs select LVRAM. The default threshold is 36 GiB.

ProfileGGUF RepositoryGGUF FileMMPROJ RepositoryMMPROJ File
HVRAMHF_MODEL_HVRAM_GGUF[1-6]HF_MODEL_HVRAM_GGUF_FILE[1-6]HF_MODEL_HVRAM_MMPROJ_GGUF[1-6]HF_MODEL_HVRAM_MMPROJ_GGUF_FILE[1-6]
LVRAMHF_MODEL_LVRAM_GGUF[1-6]HF_MODEL_LVRAM_GGUF_FILE[1-6]HF_MODEL_LVRAM_MMPROJ_GGUF[1-6]HF_MODEL_LVRAM_MMPROJ_GGUF_FILE[1-6]

Only variables belonging to the selected profile are downloaded. The existing HF_MODEL_GGUF[1-6] and HF_MMPROJ_GGUF[1-6] variables remain VRAM-independent and are always processed. Set VRAM_THRESHOLD to change the boundary.

🤖 Transformers Model Downloads

Model TypeHugging Face URL VariableDestination Subfolder VariableInclude Filter VariableExclude Filter Variable
TransformersHF_MODEL[1-6]HF_MODEL_DIR[1-6]HF_MODEL_INCLUDE[1-6]HF_MODEL_EXCLUDE[1-6]

HF_MODEL_INCLUDE[1-6] and HF_MODEL_EXCLUDE[1-6] are optional and map to hf download --include and hf download --exclude. If no destination subfolder is set, filtered files are downloaded into /workspace/textgen/user_data/models/.

Hugging Face download behavior

All Hugging Face model downloads start with the Xet backend. Before downloading, the script performs an hf download --dry-run to report the expected total size. During the transfer, RunPod logs show downloaded gigabytes and transfer speed. If the download has no output or file growth for 300 seconds, the stalled process is stopped and retried automatically with Xet disabled (plain HTTP).

Environment VariableDescriptionDefault
HF_DOWNLOAD_STALL_TIMEOUTSeconds without download activity before the current attempt is stopped300
HF_DOWNLOAD_KILL_AFTERGrace period in seconds before a stalled process is force-killed30

🤖 EXL Model Downloads

Model TypeHugging Face URL VariableRevisionDestination Subfolder Variable
EXLHF_EXL[1-6]HF_EXL_REVISION[1-6]HF_EXL_DIR[1-6]

Connection options

textgen is checked locally on port 7860 before model provisioning continues. TEXTGEN_START_MAX_TRIES controls the maximum number of checks, with five seconds between attempts; the default is 60 (about five minutes).

Services
ServicePort
Code Server9000 (HTTP)
SSH/SCP22 (TCP)
Gradio7860 (HTTP)

Website models

Websites software Github

Tag summary

Content type

Image

Digest

sha256:4b1a0c996

Size

12.6 GB

Last updated

about 1 month ago

docker pull ls250824/run-textgen:19082026