lovewin10/vllm-v14 repository overview

⁠vllm-v14

Modified ghcr.io/local-inference-lab/vllm:release-10152026⁠ by lovewin10. Not an official vLLM, FlashInfer or local-inference-lab image.

The local-inference-lab build of vLLM (V2 model runner, b12x kernels in FlashInfer) with the V14 episodic memory added: a model directory that names the V14 architecture is served with a memory that persists across requests, so a fact told in one chat is answered in a later, separate chat. It is made for LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v14⁠, whose FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales) this vLLM build serves and Transformers cannot load. CUDA graphs (FULL_AND_PIECEWISE), the prefix cache, MTP speculative decoding and async scheduling stay on; the entrypoint and the PROFILE mechanism are the base image's.

⁠What is changed

• vllm/v14 (new): the memory manager (the same code as the SGLang port's, with Triton kernels for the decode-step and prefill merges, checked against torch ops at start-up), the vLLM side of it, and a Transformers compatibility layer for the V14 configuration.
• Five files of the base image, each marked as modified:
  • vllm/transformers_utils/config.py: the V14 architecture name is mapped to the native Qwen4ExpForConditionalGeneration, so every list, profile and the MTP drafter treat it as the native model; with the memory on, the b12x QSA plans are asked for their log-sum-exp.
  • vllm/v1/worker/gpu/model_runner.py (V2 runner): the manager is built once the weights are loaded (before the memory profile), requests are registered and released, and the memory is called around each forward.
  • vllm/models/qwen4_exp/nvidia/b12x_qsa.py: recalled events are merged into each b12x QSA call by log-sum-exp, before the output gate.
  • vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py: after the b12x GDN prefill, the zero-start state of each stored event (its engram).
  • flashinfer/experimental/b12x/attention/qsa/_contract.py: with B12X_QSA_RETURN_LSE=1, plans without context parallelism return the log-sum-exp too.

⁠Run

On one 96 GB GPU (measured on an RTX PRO 6000 Blackwell), with the model repository downloaded to a folder:

docker run -d --name flash-next-v14 --gpus all --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:8000:8000 \
  -e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 \
  -v <model folder>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory \
  lovewin10/vllm-v14:release-10152026-v14

• VLLM_V14_MEMORY=0: serve the folder as the plain checkpoint, exactly as the base image does.
• VLLM_V14_MAX_SLOTS (default 4): requests that read the memory at once; about 100 MiB each, taken from the KV cache. Other requests of the same step run without memory; the ones with a slot keep theirs.
• VLLM_V14_INDEX_GB (unset by default): the store's index rows are reserved at start-up for the payloads it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB); a number pins the size.
• VLLM_V14_WORK_GB (default 1.25): GPU memory held back from the KV cache for the memory's work the vLLM profile does not see (a prefill's recurrent states, recent keys, far contexts).
• VLLM_V14_KV_CACHE_GB (default 0.25): GPU memory for recently recalled events' decoded keys; VLLM_V14_LEARN_CPU=0 fits the world model on the GPU instead of the CPU.
• VLLM_V14_TRACE=1: log every recall decision.

The GPU shows nearly full with or without the memory: the profile gives everything below --gpu-memory-utilization 0.96 to the KV cache. To leave memory free, add -e GPU_MEMORY_UTILIZATION=0.85 or a fixed -e KV_CACHE_MEMORY_BYTES=... (the base image's options).

Host memory stays level as the store grows. The image pushed before the evening of 2026-11-12 kept about 0.2 MB of host memory for every event it stored and a world model fit's copy of the index until a later garbage collection; under WSL 2 a long session could exhaust Windows' commit and stop the virtual machine. Pull the image again; the 10-minute log line gives the process's own memory apart from the PLE table in shared memory.

The store lives in /memory; delete the folder to forget. Every request reads and writes the same store: run one server with its own /memory for each user whose memories must stay apart.

⁠Measured

RTX PRO 6000 Blackwell, Docker Desktop, the command above, an empty store, every statement and question a separate chat (temperature 0, thinking off):

The model card has the details and the limits (what is recalled, two-step questions, prefills without a CUDA graph, host memory under WSL 2).

⁠Source and license

• The patch that builds this image (Dockerfile, the changed files, the originals they were made from) is in the model repository under vllm-v14-patch/; the build stops if the base image's files differ from those originals.
• vLLM and FlashInfer are Apache-2.0, and so are the patch and the memory code. Everything else in the image is the base image's, under its components' own licenses (the base is built on NVIDIA's PyTorch container).