vectorchai/scalellm repository overview

The image built with cuda 12.4⁠ is for ScaleLLM⁠: a high-performance inference system for LLMs.

Once you have Docker installed, you can run the Docker container using the following command:

docker run -it --gpus=all --net=host \
  -v $HOME/.cache/huggingface/hub:/models \
  -e HF_MODEL_ID=TheBloke/Llama-2-7B-chat-AWQ \
  -e DEVICE=cuda:0 \
  docker.io/vectorchai/scalellm:latest --logtostderr

This command starts the Docker container with GPU support and various configuration options.

• HF_MODEL_ID specifies which Hugging Face model you want to run.
• HF_MODEL_REVISION specifies which Hugging Face model revision you want to run. by default, it is set to "main".
• HF_MODEL_ALLOW_PATTERN specifies which types of files are allowed to be downloaded. by default, it is set to ".json,.safetensors,*.model".
• DEVICE specifies the device on which this model should run. by default, it is set to "auto".
• HUGGING_FACE_HUB_TOKEN specifies the token from huggingface for gated models.

After running the Docker container, two ports are exposed:

1: Port 8888 for gRPC Server:

The gRPC server is served on 0.0.0.0:8888 by default. You can use gRPC to interact with the service.

2: Port 9999 for HTTP Server:

The simple HTTP server for instrument will be served on 0.0.0.0:9999 by default. This server provides various endpoints for managing and monitoring the service:

• Use curl localhost:9999/health to check the health status of the service.
• Use curl localhost:9999/metrics to export Prometheus metrics.
• Use curl localhost:9999/gflags to list all available gflags for configuration.
• add more to come...