intel/vllm repository overview

⁠Contents of the vLLM Container

This container image is optimized for use with Intel® GPUs. It includes the complete source for this vLLM release in /opt/vllm, and vLLM is installed in the Python virtual environment at /opt/venv (package path: /opt/venv/lib/python3.12/site-packages/vllm). It has been validated on Intel® Arc™ Pro B-Series Graphics⁠. The following bill of materials was used during validation:

For more information about vLLM, visit vLLM Docs⁠.

⁠Dockerfiles

The Dockerfiles used to build the container images are available by the links below:

• 0.21.0-ubuntu24.04.dockerfile⁠

⁠1. What's New in This Release?

⁠vLLM 0.21.0

• Most workloads achieve obvious performance improvement compared with v0.14.1⁠ release, especially in 1k/512 input/output seq length scenarios with up to 1.74X boost. Note that in this release, the first round performance doesn't reflect peak values because of JIT compilation overhead. It is recommended to perform warm-up runs before and set --temperature=0 to collect stable and reproducible performance data.
• MLA (Multi-head Latent Attention) is functionally supported and validated with the deepseek-ai/DeepSeek-V2-Lite model.
• GDN (GatedDeltaNet) attention XPU kernel is introduced for Qwen3-Next models, delivering more than 1.1x end-to-end performance improvement compared with the Triton implementation.

⁠2. What's Supported?

This release supports core vLLM serving capabilities on Intel® GPUs, including online FP8 quantization, multimodal models, pooling models, and multi-GPU scaling strategies. In addition to dense model serving, it also includes expert parallelism and validated support for MoE models.

In addition, features such as async scheduling, cpu kv cache offloading, automatic prefix caching⁠, reasoning outputs⁠, structured outputs⁠ and tool calling⁠ are also supported.

⁠3. Supported Models

The following tables list the models validated by Intel. Support for vLLM on Intel® GPUs extends to a broader set of models beyond this list.

⁠Text Generation Models

These models primarily use the LLM.generate API. Chat and instruct variants also support the LLM.chat API.

Per-tensor and per-channel compressed_tensors quantized models such as RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic are also supported.

⁠Multimodal Models

Supported modalities vary by model and include text, image, video, and audio:

⁠Pooling Models

These models primarily support the LLM.embed API. The following table lists the models validated on XPU.

⁠4. Limitations

The following items are currently known issues and limitations:

• vllm serve uses aggressive GPU memory allocation by default. If you encounter an out-of-memory (OOM) condition, reduce the GPU memory utilization setting when starting the server. For example: vllm serve --gpu-memory-utilization 0.8.
• torch.compile is currently not supported.
• For BF16 models with 8K input and 2K output sequence lengths, TPOT performance shows an approximately 10% regression compared to the 0.14.1 release. This issue is expected to be resolved with PyTorch 2.12.
• A TPOT performance regression has been observed for the Qwen/Qwen3-30B-A3B model with 8K input and 2K output sequence lengths. The regression is caused by additional host overhead introduced by the vLLM FusedMoE refactor in upstream.

⁠5. How to Get Started

⁠5.1. Prerequisites

⁠5.2. Prepare a Serving Environment

1. Pull the released Docker image:

   docker pull <docker-image-name>:<tag>
   

   Copy

2. Start the container:

   docker run -t -d --shm-size 10g --net=host --ipc=host --privileged \
     -v /dev/dri/by-path:/dev/dri/by-path --name=vllm-test \
     --device /dev/dri:/dev/dri --entrypoint=/bin/bash <docker-image-name>:<tag>
   

   Copy

3. Open two terminals and run docker exec -it vllm-test bash in both. Use one terminal for the server and the other for the client.

From this point on, all commands are expected to be run inside the Docker container unless noted otherwise.

In both environments, you may want to set the HUGGING_FACE_HUB_TOKEN environment variable so the required files can be downloaded from Hugging Face.

export HUGGING_FACE_HUB_TOKEN=xxxxxx

⁠5.3. Launch Workloads

⁠5.3.1. Launch the Server in the Server Environment

Command:

VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --dtype=float16 \
  --enforce-eager \
  --port 8000 \
  --block-size 64 \
  --gpu-memory-utilization 0.9 \
  --no-enable-prefix-caching \
  --trust-remote-code \
  --max-num-batched-tokens=8192 \
  --max-model-len 4096 \
  -tp=4 \
  --quantization fp8

Expected output:

INFO 03-20 03:20:29 api_server.py:937] Starting vLLM API server on http://0.0.0.0:8000
INFO 03-20 03:20:29 launcher.py:23] Available routes are:
INFO 03-20 03:20:29 launcher.py:31] Route: /openapi.json, Methods: HEAD, GET
INFO 03-20 03:20:29 launcher.py:31] Route: /docs, Methods: HEAD, GET
INFO 03-20 03:20:29 launcher.py:31] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 03-20 03:20:29 launcher.py:31] Route: /redoc, Methods: HEAD, GET
INFO 03-20 03:20:29 launcher.py:31] Route: /health, Methods: GET
INFO 03-20 03:20:29 launcher.py:31] Route: /ping, Methods: POST, GET
INFO 03-20 03:20:29 launcher.py:31] Route: /tokenize, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /detokenize, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/models, Methods: GET
INFO 03-20 03:20:29 launcher.py:31] Route: /version, Methods: GET
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/chat/completions, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/completions, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/embeddings, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /pooling, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /score, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/score, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /rerank, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v1/rerank, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /v2/rerank, Methods: POST
INFO 03-20 03:20:29 launcher.py:31] Route: /invocations, Methods: POST
INFO:     Started server process [1636943]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Startup may take some time. When INFO: Application startup complete. appears, the server is ready.

⁠5.3.2. Send Benchmark Requests from the Client Environment

Use the following command to send benchmark requests:

vllm bench serve \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --dataset-name random \
  --random-input-len=1024 \
  --random-output-len=1024 \
  --ignore-eos \
  --num-prompt 16 \
  --max-concurrency 16 \
  --temperature=0 \
  --request-rate inf \
  --backend vllm \
  --port=8000 \
  --host 0.0.0.0 \
  --ready-check-timeout-sec 1

This command uses the deepseek-ai/DeepSeek-R1-Distill-Qwen-32B model. Both the input and output lengths are set to 1024 tokens, and the server processes up to 16 requests concurrently.

Expected output:

Maximum request concurrency: 16
============ Serving Benchmark Result ============
Successful requests:                     1
Benchmark duration (s):                  xxx
Total input tokens:                      1024
Total generated tokens:                  1024
Request throughput (req/s):              xxx
Output token throughput (tok/s):         xxx
Total Token throughput (tok/s):          xxx
---------------Time to First Token----------------
Mean TTFT (ms):                          xxx
Median TTFT (ms):                        xxx
P99 TTFT (ms):                           xxx
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          xxx
Median TPOT (ms):                        xxx
P99 TPOT (ms):                           xxx
---------------Inter-token Latency----------------
Mean ITL (ms):                           xxx
Median ITL (ms):                         xxx
P99 ITL (ms):                            xxx
==================================================

⁠Need Assistance?

If you encounter any issues or have questions, please submit an issue at vLLM GitHub Issues⁠. Include the text [Intel GPU] in the issue title so it is routed appropriately.

⁠License agreement

By downloading and using these container images and the included software, you agree to the terms and conditions of the software license agreements⁠.

In accordance with the terms and conditions of these licenses, particularly those requiring source code availability, such as GPL, the source code for all open-source components included in these container images can be obtained from here⁠.

⁠Notes

All Ubuntu container images are intended for demo purposes only and not intended for production use. To receive expanded security maintenance from Canonical on the Ubuntu base layer, you may follow the how-to guide to enable Ubuntu Pro in a Dockerfile⁠ which will require the image to be rebuilt.

FFmpeg is an open source project licensed under LGPL and GPL. See https://www.ffmpeg.org/legal.html⁠. You are solely responsible for determining if your use of FFmpeg requires any additional licenses. Intel is not responsible for obtaining any such licenses, nor liable for any licensing fees due, in connection with your use of FFmpeg.

⁠Acknowledgements

© Intel Corporation. Intel, Intel® Graphics Compute Runtime for oneAPI Level Zero and OpenCL™ Driver and other Intel marks are trademarks of Intel Corporation or its subsidiaries.

* Other names and brands may be claimed as the property of others.