tantransiliconstack/comfyui-vulkan repository overview

⁠comfyui-vulkan

ComfyUI on PyTorch's Vulkan backend: it can use any GPU that has a Vulkan driver, including cards that ROCm no longer supports, such as AMD gfx906 / MI50 / Radeon VII. It also runs on Intel and NVIDIA.

⁠Quick start

Image: tantransiliconstack/comfyui-vulkan⁠ on Docker Hub. Tags: 0.2.0-fp16 (= latest; model-aware VRAM cache), 0.1.0-fp16 (older); linux/amd64.

AMD or Intel GPU on Linux. You need the kernel amdgpu/i915/xe driver loaded (ls /dev/dri shows renderD128). No ROCm is needed: the Vulkan driver (Mesa RADV/ANV) is inside the image.

mkdir -p ~/comfy-data/{input,output,user,custom_nodes,temp} ~/comfy-data/models/{diffusion_models,text_encoders,vae,loras,checkpoints}
docker run -d --name comfyui-vulkan \
  --device /dev/dri \
  --group-add "$(getent group video | cut -d: -f3)" --group-add "$(getent group render | cut -d: -f3)" \
  -p 127.0.0.1:8188:8188 -v ~/comfy-data:/data \
  tantransiliconstack/comfyui-vulkan

Put your models in ~/comfy-data/models/... (same layout as ComfyUI's models/ folder), then open http://127.0.0.1:8188⁠. For a remote server, open a tunnel first: ssh -L 8188:127.0.0.1:8188 user@server. Port 8188 is bound to 127.0.0.1 on purpose: ComfyUI has no login. The container runs as UID 1000; if ~/comfy-data belongs to someone else, add --user "$(id -u):$(id -g)".

• NVIDIA: replace --device /dev/dri ... with --gpus all. The image already asks the NVIDIA toolkit for its Vulkan driver (NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics).
• Docker Desktop on Windows/macOS: no GPU reaches a Linux container this way, so the image runs pure CPU. The only GPU route on Windows, Mesa's D3D12 driver "dzn", returned wrong numbers in our tests and is not shipped.

⁠Check what it found (30 seconds, no model needed)

docker run --rm --device /dev/dri --group-add "$(getent group video | cut -d: -f3)" --group-add "$(getent group render | cut -d: -f3)" tantransiliconstack/comfyui-vulkan python -m comfyui_vulkan

The output has two parts:

• Human-readable summary (stderr): which GPU was chosen, whether tensors are stored as fp16 or fp32, and for each kind of math whether it passed the correctness self-test and how its benchmark compared with the CPU.
• JSON (stdout): please attach it to bug reports. It is the measurement this project needs most.

Good: the device: line names your GPU. Bad: it says llvmpipe ... (CPU), which means the GPU did not reach the container. Check --device /dev/dri and the group IDs.

⁠Settings

After every prompt the log prints a line [comfyui-vulkan] calls in this prompt: {...}. It counts, per kind of math, how many calls ran on the GPU, how many fell back to the CPU and why, and the weight-cache hits.

⁠How it works (and what happens when a step fails)

ComfyUI itself runs its normal CPU code path and never sees a GPU tensor. Three torch functions (F.linear, F.conv2d, F.scaled_dot_product_attention) and ComfyUI's quantized Linear layer are wrapped. Each wrapped call sends its data to the GPU, computes there, and returns an ordinary CPU tensor. Any call it cannot handle simply runs the original CPU code.

Anchors are file:line in this source tree, checked 2026-10-21.

container start: python main.py --vulkan ...
 └─ --vulkan implies --cpu, no pinned memory, PyTorch attention ........ patches/comfyui/0001 (comfy/cli_args.py)
 └─ comfyui_vulkan.init(mode) ........................................... comfyui_vulkan/__init__.py:932 init
     ├─ probe(): is Vulkan available? which device? its limits ......... __init__.py:120 probe, patch 0001 device_info()
     │    ├─ no driver / no GPU in the container → "Vulkan unavailable" → everything runs on CPU (slow, still works)
     │    ├─ device chosen = first GPU, or PYTORCH_VULKAN_DEVICE ......... patch 0001 select_from_env()
     │    └─ chosen device is a CPU emulation (llvmpipe) → banner warns "no GPU reached the container"
     ├─ storage check: is this build fp16 or fp32? ..................... __init__.py:143 _detect_storage
     ├─ self-test each kind (linear, conv2d, attention) vs CPU ........ __init__.py:831 _cases
     │    └─ wrong result or error on YOUR driver → that kind stays on CPU; the reason is in the banner
     ├─ weight cache budget = GPU memory − reserve − slack ............. __init__.py:182 _cache_budget_from
     │    ├─ patched torch: live VRAM budget (VK_EXT_memory_budget), VRAM-only cache fill,
     │    │  no CPU copy of cached weights ............................. __init__.py:889 _setup_memory_ops
     │    │  (older torch / driver without the extension → static heap size; the banner says which)
     │    ├─ cache self-test fails → cache off ......................... __init__.py:873 _cache_selftest
     │    └─ ok → listen to ComfyUI's load_models_gpu and unload_all_models ... owners.py:140 install
     ├─ quantized Linear (QLinear) self-test + hook into ComfyUI ...... comfyui_vulkan/qlinear.py:253 setup
     │    ├─ torch build without the op → qlinear off
     │    └─ ComfyUI internals changed (signature check) → hook not installed, ComfyUI unchanged
     └─ auto mode: benchmark each kind vs CPU → on only if faster ...... __init__.py:932 init

each node that needs a model (text encode, sampler, VAE decode):
 ComfyUI calls load_models_gpu([model]) → recorded as "needed now" ..... owners.py:82 note
   ├─ a "restore all" call (every loaded model again, e.g. ControlNet + VAE mid-sampling) only
   │  refreshes recency; "needed now" goes back to what it was before
   └─ models ComfyUI has dropped (garbage-collected) → their cache entries freed now ... __init__.py:241 _on_note
 ComfyUI "unload models" (unload_all_models) → the whole weight cache is freed ... __init__.py:252 _on_unload_all

each call during a prompt:
 dense Linear → F.linear wrapper ....................................... __init__.py:764 linear
   → tiles so no dimension exceeds the GPU's limit AND one step fits the reserve ... __init__.py:466 _linear_steps, __init__.py:500 vk_linear
   → weight from cache or uploaded ...................................... __init__.py:395 _cached_grid
      ├─ cache full → evict whole models NOT needed now, oldest request first; never the
      │  model being run (inside one model: keep what fits) ................ __init__.py:274 _make_room
      ├─ the cache never exceeds the driver's live budget (re-asked ≤ every 100 ms) ... __init__.py:326 _live_cap
      └─ filling with VRAM-only allocations; VRAM full → layer not cached, runs
         uncached on the GPU (temporary buffers may spill to RAM, never fail) ... __init__.py:352 _fill_device_local
 fp8 / int8 quantized Linear → MixedPrecisionOps.Linear.forward hook ....... qlinear.py:183 _patch
   → raw 8-bit bytes + scale → QLinear op on the GPU (1 byte/weight) ...... qlinear.py:146 vk_qlinear, torch-vulkan-qlinear/.../ops/QLinear.cpp:105
   ├─ LoRA weight function, AWQ, forced cast, gradients → ComfyUI's own forward (counted "cpu: <reason>")
   └─ e5m2 / nvfp4 / mxfp8 / convrot / transposed int8 → ComfyUI's own forward ("cpu: unsupported layout")
 Conv2d → __init__.py:780 conv2d (groups=1, no dilation, fits the limit AND the reserve)   else CPU
 attention → __init__.py:795 scaled_dot_product_attention (no mask, not causal), tiled to the reserve ... __init__.py:561 _attn_steps   else CPU
 every GPU call goes through _run ...................................... __init__.py:637 _run
   ├─ fp16 build and an input ≥ 65504 → CPU (it would be clamped on upload)
   ├─ result ≥ 65504 (fp16 clamp) or inf/NaN → recomputed on CPU
   ├─ out of VRAM → evict other models (or cap the cache until the prompt ends), flush, retry (≤2×),
   │  then CPU; never switches the kind off ................................ __init__.py:732 _relieve
   ├─ out of host RAM → retry, then CPU; the VRAM cache is left alone
   ├─ device lost → every kind switched off
   └─ other exception → CPU; 3 exceptions of one kind → that kind is switched off and its cache entries freed
after each prompt: "[comfyui-vulkan] calls in this prompt: {...}" ...... patches/comfyui/0001 (main.py)
   └─ also clears a temporary cache cap and purges dropped models .......... __init__.py:1059 take_stats

If something looks wrong, look in this order:

1. The startup banner: which device, which kinds are ON, and why the others are off.
2. The per-prompt calls line: what ran where, and why.
3. python -m comfyui_vulkan force: it re-runs the self-tests and prints the JSON.

⁠Where state lives

⁠What it does NOT do

• It does not make ComfyUI a "GPU" ComfyUI. Weights live in system RAM and the GPU only gets the math, so you need system RAM for the model (for example ~24 GB for Krea-2 Turbo fp8).
• It does not cache weights in the default mode on integrated GPUs or CPU devices (they share RAM).
• It only knows how much VRAM is free if the driver supports VK_EXT_memory_budget (the banner line "VRAM source" says). Without it, another program using the same GPU is not seen. Either way a full VRAM never fails a prompt: cached weights must fit in VRAM or are simply not cached; temporary buffers may spill to system RAM (slower, still computed on the GPU); an out-of-memory error evicts, caps the cache until the prompt ends and retries, then uses the CPU. "VRAM-only" is a request for VRAM memory types: the Linux kernel driver can still move a buffer to system RAM under pressure, so the live budget, not the request, is the main guard.
• It does not tile convolutions spatially: a Conv2d whose single slice needs more than the reserve runs on the CPU.
• These run on the CPU:
  • Conv3d, e.g. the Wan/Qwen VAE;
  • masked or causal attention;
  • grouped/dilated convolutions;
  • anything larger than the GPU's texture limit;
  • quantized formats other than fp8 e4m3fn and int8 tensorwise/rowwise.
• int8 layers compute with fp32 activations (weight-only). ComfyUI's own CPU path quantizes activations too, so int8 output differs slightly from the CPU image, in the more precise direction.
• It does not tune kernels for any GPU yet. PyTorch's Vulkan matmul and our 8-bit kernel are simple, untuned versions.
• It does not guarantee bit-identical images with the default fp16 build: Krea-2 measured PSNR 31.5–32.5 dB vs CPU, visually the same in our test. Build with --build-arg VULKAN_FP16=0 --build-arg TORCH_LOCAL_VERSION=vulkan for ~66 dB at twice the GPU memory.
• It does not speed up LLM-style token-by-token generation (for example the "TextGenerate" prompt-expansion node) unless those weights fit in the cache.

⁠Tested on

⁠Build from source

docker build -t comfyui-vulkan .                                                          # fp16 storage (default)
docker build --build-arg VULKAN_FP16=0 --build-arg TORCH_LOCAL_VERSION=vulkan -t comfyui-vulkan:fp32 .

The build compiles PyTorch 2.14.1 with USE_VULKAN=1 (CPU-only otherwise) and a matching torchvision. It applies patches/pytorch/, adds the QLinear op from torch-vulkan-qlinear/, and patches ComfyUI (pinned commit) with patches/comfyui/. A first build takes about 40–60 minutes on 20 threads; later builds reuse the compiler cache.

Tests (run inside the image with a Vulkan driver available, e.g. -e VK_DRIVER_FILES=/usr/share/vulkan/icd.d/lvp_icd.json):

• tests/vk_probe.py: probes the driver.
• tests/qlinear_op_test.py: tests the QLinear op.
• tests/qlinear_hook_test.py: tests the ComfyUI hook.
• tests/cache_test.py: tests the weight cache.
• tests/overflow_test.py: tests the fp16 overflow guard.
• tests/cache_evict_test.py: tests model-aware eviction, out-of-VRAM retry, the per-op reserve and the load_models_gpu listener (needs -e PYTHONPATH=/opt/ComfyUI).

⁠Licenses

ComfyUI is GPL-3.0; PyTorch and torchvision are BSD-3-Clause; Mesa is MIT. See NOTICE.md (also in the image at /opt/comfyui-vulkan/NOTICE.md). All modifications ship as source inside the image:

• /opt/comfyui-vulkan/patches/
• /opt/comfyui-vulkan/torch-vulkan-qlinear/
• /opt/ComfyUI/comfyui_vulkan/