zentorch provides optimized support for Large Language Model (LLM) inference through integration with vLLM. This repository hosts the official Docker images for vLLM–zentorch releases, enabling users to run high‑performance LLM inference in a consistent and reproducible containerized environment.
The Docker images in this repository are built specifically to support LLM inference workflows using zentorch and are preconfigured with settings recommended for production and benchmarking scenarios.
The Docker images are available for the following operating systems:
| Operating System | Version |
|---|---|
| Ubuntu | 22.04 |
| RHEL | 9.5 |
Each image is built and tested to ensure compatibility with the corresponding OS base image and runtime dependencies required by vLLM and zentorch.
For the latest docker releases, please refer to vLLM-zentorch Docker releases
The Docker images include recommended default configurations for LLM inference, such as:
# Pull and run the image
docker pull amdih/zendnn_zentorch:vllm_v0.13.0_zentorch_v5.2.0_ubuntu22.04_2026_ww02
docker run -it --privileged -d --name vllm_zentorch_ubuntu22 amdih/zendnn_zentorch:vllm_v0.13.0_zentorch_v5.2.0_ubuntu22.04_2026_ww02
# Run the container
docker exec -it vllm_zentorch_ubuntu22 bash
# Run the model
# Offline benchmarking
vllm bench throughput --model meta-llama/Llama-3.1-8B-Instruct --input-len 128 --output-len 128 --num-prompts 512 --max-num-seqs 32
# Online benchmarking
# Serve the model
vllm serve OpenGVLab/InternVL3-2B --disable-log-requests --enable-chunked-prefill --enable-prefix-caching --trust_remote_code --host 127.0.0.1 --port 8000
# Benchmark the model
vllm bench serve --model OpenGVLab/InternVL3-2B --endpoint /v1/completions --dataset-name random --percentile-metrics tpot,ttft,itl,e2el --metric-percentiles 50,90,95,99 --random-input-len 128 --random-output-len 1 --num-prompts 200
# Stop the container
docker stop vllm_zentorch_ubuntu22
# Remove the container
docker rm vllm_zentorch_ubuntu22
# RHEL 9.5
docker pull amdih/zendnn_zentorch:vllm_v0.13.0_zentorch_v5.2.0_rhel9.5_2026_ww02
docker run -it --privileged -d --name vllm_zentorch_rhel9.5 amdih/zendnn_zentorch:vllm_v0.13.0_zentorch_v5.2.0_rhel9.5_2026_ww02
# Run the container
docker exec -it vllm_zentorch_rhel9.5 bash
# Run the model
# Offline benchmarking
vllm bench throughput --model meta-llama/Llama-3.1-8B-Instruct --input-len 128 --output-len 128 --num-prompts 512 --max-num-seqs 32
# Online benchmarking
# Serve the model
vllm serve OpenGVLab/InternVL3-2B --disable-log-requests --enable-chunked-prefill --enable-prefix-caching --trust_remote_code --host 127.0.0.1 --port 8000
# Benchmark the model
vllm bench serve --model OpenGVLab/InternVL3-2B --endpoint /v1/completions --dataset-name random --percentile-metrics tpot,ttft,itl,e2el --metric-percentiles 50,90,95,99 --random-input-len 128 --random-output-len 1 --num-prompts 200
# Stop the container
docker stop vllm_zentorch_rhel9.5
# Remove the container
docker rm vllm_zentorch_rhel9.5
Content type
Image
Digest
sha256:d4f81496c…
Size
4 GB
Last updated
1 day ago
docker pull amdih/zendnn_zentorch:vllm_v0.31.0_zentorch_v2.13.0.3_rhel10.2_2026_ww41