darkkernel/vllm-gfx1030 repository overview

⁠vLLM for AMD gfx1030 (RDNA2 / Navi 21)

Ready-to-run vLLM⁠ OpenAI-compatible inference server built for AMD gfx1030 GPUs — Navi 21 / RDNA2, e.g. Radeon Pro V620, RX 6800 / 6800 XT / 6900 XT / 6950 XT.

The official rocm/vllm-dev images only ship kernels for CDNA (gfx90a/942), RDNA3 (gfx1100/1101) and RDNA4 (gfx120x) — not gfx1030 — so they crash on these cards with HIP error: invalid device function. This image is compiled specifically for gfx1030 and works out of the box.

⁠Tags

⁠Quick start

docker run -d --name vllm-server \
  --device=/dev/kfd --device=/dev/dri \
  --security-opt seccomp=unconfined \
  --ipc=host --shm-size=16g \
  -e HIP_VISIBLE_DEVICES=0 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -p 8000:8000 \
  darkkernel/vllm-gfx1030:v0.19.1-rocm7.2 \
  vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --max-model-len 8192 --host 0.0.0.0 --port 8000

Then call the OpenAI-compatible API:

curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
       "prompt":"The capital of France is","max_tokens":24}'

⁠Requirements

• An AMD gfx1030 GPU (Navi 21 / RDNA2).
• Host with the AMDGPU / ROCm kernel driver and access to /dev/kfd + /dev/dri (pass both with --device).
• No --group-add needed if the render nodes are accessible; otherwise add the host's render/video group.

⁠Multi-GPU

For larger models across multiple cards:

  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  ... vllm serve <model> --tensor-parallel-size 4 ...

(The custom all-reduce kernel is auto-disabled on Navi; vLLM falls back automatically.)

⁠What's inside / how it was built

Built from source on rocm/pytorch:latest (whose PyTorch does include gfx1030) with PYTORCH_ROCM_ARCH=gfx1030.

Two things that make gfx1030 work and are baked into this image:

1. Version pairing. rocm/pytorch:latest ships torch 2.10.0+rocm7.2.4. vLLM main/v0.20+ pin torch 2.11 and fail to compile against 2.10 (torch::stable::Tensor API drift). vLLM v0.19.1 pins torch 2.10.0 and builds cleanly — that's the version here.
2. RDNA2 shared-memory fix. RDNA2 has only 64 KB LDS per workgroup. The Triton prefix-prefill attention kernel's default 128×64 tile with head_dim 128 needs ~80 KB and dies at inference with triton.runtime.errors.OutOfResources: shared memory. This image patches the tile down to 32×32 on gfx10xx/gfx11xx.

⁠Notes

• A harmless ImportError: cannot import name 'RoutedExperts' may appear at startup — it's an amd-quark quantization plugin probe and does not affect serving non-quantized models.
• Model advertised context is often very large (e.g. 131072); keep --max-model-len modest (e.g. 8192) on a single GPU to avoid OOM during memory profiling.

⁠License

vLLM is Apache-2.0. This image redistributes vLLM, PyTorch, and ROCm components under their respective licenses.