slinusc/deepspeed-mii repository overview

⁠Overview

This repository provides instructions for running a DeepSpeed-MII OpenAI-Compatible server via a prebuilt image on Docker Hub. This community image allows you to deploy and serve any Hugging Face model (e.g., mistralai/Mistral-7B-Instruct-v0.3) with an OpenAI-API–compatible interface.

⁠Table of Contents

1. Prerequisites⁠
2. Docker Hub Image⁠
3. Running the Container⁠
4. Testing with curl⁠
5. Testing with Python + OpenAI SDK⁠
6. Environment Variables⁠
7. Customizing & Troubleshooting⁠
8. License⁠

⁠Prerequisites

• Docker (20.10+).

• NVIDIA Container Toolkit (to allow --gpus all).

• (Optional) OpenAI Python SDK for Python-based testing:

  pip install openai
  

  Copy

• A valid Hugging Face Hub token if you plan to load private or gated models:

  export HF_TOKEN=<your_hf_token>
  

  Copy

⁠Docker Hub Image

Because Microsoft does not provide an official DeepSpeed-MII Docker image, this repository relies on a community-maintained image hosted on Docker Hub. To pull the latest version:

docker pull slinusc/deepspeed-mii:latest

• Image name: slinusc/deepspeed-mii
• Tag: latest

Once pulled, you can skip the “build from source” steps and jump directly to Running the Container⁠.

⁠Running the Container

Use the prebuilt image to launch DeepSpeed-MII with GPU support, mount your Hugging Face cache, and expose port 23333:

docker run --gpus all \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
  -p 127.0.0.1:23333:23333 \
  --ipc=host \
  slinusc/deepspeed-mii:latest \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --port 23333

Flags explained:

• --gpus all – Grant the container access to all available GPUs.
• -v $HOME/.cache/huggingface:/root/.cache/huggingface – Mount your local HF cache so model weights aren’t re-downloaded.
• -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN – Pass your HF token into the container.
• -p 127.0.0.1:23333:23333 – Map container port 23333 → host 23333.
• --ipc=host – Share IPC namespace to reduce overhead.
• slinusc/deepspeed-mii:latest – The Docker Hub image name and tag.
• --model mistralai/Mistral-7B-Instruct-v0.3 – Specify the HF model to load on startup.
• --port 23333 – Force Uvicorn inside the container to bind to port 23333.

After running, you should see logs similar to:

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:23333 (Press CTRL+C to quit)

The server is now live at http://127.0.0.1:23333/v1/....

⁠Testing with curl

Once the container is running in the background, open a new terminal window/tab and run the following commands:

1. List available models

   curl http://127.0.0.1:23333/v1/models
   

   Copy

   Expected JSON response:

   {
     "object": "list",
     "data": [
       {
         "id": "mistralai/Mistral-7B-Instruct-v0.3",
         "object": "model",
         "created": 1748684820,
         "owned_by": "deepspeed-mii",
         "root": "mistralai/Mistral-7B-Instruct-v0.3",
         "parent": null,
         "permission": [ … ]
       }
     ]
   }
   

   Copy

   The "id" field is the model name you’ll use for subsequent endpoints.

2. Chat completion request

   curl http://127.0.0.1:23333/v1/chat/completions \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{
           "model": "mistralai/Mistral-7B-Instruct-v0.3",
           "messages": [
             { "role": "system", "content": "You are a helpful assistant." },
             { "role": "user",   "content": "Tell me a fun fact about penguins." }
           ],
           "max_tokens": 32,
           "temperature": 0.7
         }'
   

   Copy

3. Text completion request (optional)

   curl http://127.0.0.1:23333/v1/completions \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{
           "model": "mistralai/Mistral-7B-Instruct-v0.3",
           "prompt": "Once upon a time in a distant galaxy,",
           "max_tokens": 50,
           "temperature": 0.7
         }'
   

   Copy

Each request returns a JSON response containing a choices array with generated output.

⁠Testing with Python + OpenAI SDK

If you prefer testing from Python, install the OpenAI SDK locally (if not already done):

pip install openai

Save this snippet as test_mii.py:

from openai import OpenAI

# Point the SDK to your local MII endpoint
client = OpenAI(
    api_key="",  # no key needed if container is in no-auth mode
    base_url="http://127.0.0.1:23333/v1"
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "How many wings does a penguin have?"}
    ],
    max_tokens=16,
    temperature=0.7
)
print(response.choices[0].message.content)

Then run:

python3 test_mii.py

If successful, you’ll see a brief answer about penguins. If it fails, double-check that:

1. Container is running
2. Correct model ID was provided
3. Port 23333 is properly mapped

⁠Environment Variables

• HF_TOKEN (host) or HUGGING_FACE_HUB_TOKEN (inside container)

  • Your Hugging Face Hub token for accessing private or gated models.

  export HF_TOKEN=<your_hf_token>
  

  Copy

• OPENAI_API_KEY (optional)

  • If you configure the container to require an API key, set this on your host and pass it into Docker:

    docker run -e OPENAI_API_KEY=<your_key> … slinusc/deepspeed-mii:latest …
    

    Copy

  • If omitted, the container runs in no-auth mode by default.

⁠Customizing & Troubleshooting

⁠Change the CUDA Base Image

If your GPU driver requires a different CUDA version, you can rebuild from source using a modified Dockerfile. Swap the FROM line:

FROM nvidia/cuda:11.8-devel-ubuntu20.04

…or any other suitable tag for your environment.

⁠Use a Different Hugging Face Model

When launching the container, simply change the --model argument:

docker run … slinusc/deepspeed-mii:latest \
  --model your-org/your-model-name \
  --port 23333

For quantized checkpoints, append a quantization flag (e.g., --quantize gptq).

⁠Common Issues

1. Container won’t start / hangs on startup

   • Ensure your CUDA driver and NVIDIA Container Toolkit versions are compatible.

   • Check Docker logs with:

     docker logs <container_id>
     

     Copy

2. Model fails to load

   • Verify that HF_TOKEN is correct and has permission to access the model.
   • Confirm you have enough GPU memory (8 GB+ recommended for 7B models).

3. Port 23333 conflicts

   • Change host port mapping, e.g., -p 127.0.0.1:12345:23333, then use http://127.0.0.1:12345/v1/....

⁠License

This repository and Docker Hub image are licensed under the MIT License. See LICENSE⁠ for details. If you do not include your own LICENSE file, usage defaults to “All rights reserved.”

Congratulations! You now have a fully functional Docker container—pulled from Docker Hub—that runs DeepSpeed-MII in OpenAI-API compatibility mode. Anyone can simply pull slinusc/deepspeed-mii:latest to run a local inference server for Mistral or any other Hugging Face model.