Modified ghcr.io/local-inference-lab/vllm:release-10152026
by lovewin10. Not an official vLLM, FlashInfer or local-inference-lab image.
The local-inference-lab build of vLLM (V2 model runner, b12x kernels in FlashInfer) with the V15 episodic
memory added: a model directory that names the V15 architecture is served with a memory that persists
across requests, so a fact told in one chat is answered in a later, separate chat, and while the server is
idle the memory sleeps: the served model itself replays the stored events, merges repeated ones, retires
the ones a later event corrects and bridges related ones (no training, no new content). It is made for
LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15,
whose FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales) this vLLM build
serves and Transformers cannot load. CUDA graphs (FULL_AND_PIECEWISE), the prefix cache, MTP speculative
decoding and async scheduling stay on; the entrypoint and the PROFILE mechanism are the base image's.
vllm/v15 (new): the memory manager and its sleep (the same code as the SGLang port's, with Triton
kernels for the decode-step and prefill merges, checked against torch ops at start-up), the vLLM side of
it (the sleep reaches the model through the server's own OpenAI API), and a Transformers compatibility
layer for the V15 configuration.vllm/transformers_utils/config.py: the V15 architecture name is mapped to the native
Qwen4ExpForConditionalGeneration, so every list, profile and the MTP drafter treat it as the native
model; with the memory on, the b12x QSA plans are asked for their log-sum-exp.vllm/v1/worker/gpu/model_runner.py (V2 runner): the manager is built once the weights are loaded
(before the memory profile) and its sleep started, requests are registered and released, and the
memory is called around each forward.vllm/models/qwen4_exp/nvidia/b12x_qsa.py: recalled events are merged into each b12x QSA call by
log-sum-exp, before the output gate.vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py: after the b12x GDN prefill, the
zero-start state of each stored event (its engram).flashinfer/experimental/b12x/attention/qsa/_contract.py: with B12X_QSA_RETURN_LSE=1, plans without
context parallelism return the log-sum-exp too.On one 96 GB GPU (an RTX PRO 6000 Blackwell), with the model repository downloaded to a folder:
docker run -d --name flash-next-v15 --gpus all --ipc=host --shm-size 32g \
--ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:8000:8000 \
-e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 \
-v <model folder>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory \
lovewin10/vllm-v15:release-10152026-v15
VLLM_V15_MEMORY=0: serve the folder as the plain checkpoint, exactly as the base image does.VLLM_V15_SLEEP=0: the memory without its sleep; VLLM_V15_SLEEP_AFTER (default 600): seconds without a
user request before a sleep cycle starts. The sleep calls the server at http://127.0.0.1:$PORT
(VLLM_V15_SELF_URL to change it; an API key as VLLM_API_KEY). Every cycle logs a V15 sleep: line.VLLM_V15_MAX_SLOTS (default: --max-num-seqs, 16 with this profile): requests that read the memory at
once, so every request the engine runs has its memory; about 100 MiB each, taken from the KV cache. A
smaller number saves that memory; requests beyond it then run without memory.VLLM_V15_INDEX_GB (unset by default): the store's index rows are reserved at start-up for the payloads
it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB); a number pins the size.VLLM_V15_WORK_GB (default 1.25): GPU memory held back from the KV cache for the memory's work the
vLLM profile does not see (a prefill's recurrent states, recent keys, far contexts).VLLM_V15_KV_CACHE_GB (default 0.25): GPU memory for recently recalled events' decoded keys.VLLM_V15_TRACE=1: log every recall decision.The GPU shows nearly full with or without the memory: the profile gives everything below
--gpu-memory-utilization 0.96 to the KV cache. A lower -e GPU_MEMORY_UTILIZATION (or a fixed
-e KV_CACHE_MEMORY_BYTES=..., the base image's options) leaves GPU memory free, all of it taken from the KV
cache: about 0.96 GiB, some 52,000 tokens, for every 0.01. Below about 0.90 one 262,144-token request no
longer fits and vLLM stops after loading the weights, so add -e MAX_MODEL_LEN=-1 as well (vLLM serves the
longest context that fits and logs it as Auto-fit max_model_len) or a smaller MAX_MODEL_LEN.
The store lives in /memory; delete the folder to forget. Every request reads and writes the same store:
run one server with its own /memory for each user whose memories must stay apart. A V14 store is not
read: V15 opens a store of its own.
One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, this image as it is with
LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15,
the profile above, an empty store and -e VLLM_V15_SLEEP_AFTER=20; every statement and every question a
chat of its own, temperature 0, thinking off. A handful of questions per check, not a benchmark.
| Memory on | Memory off (the base engine) | |
|---|---|---|
| A fact told in one chat, asked in a new chat | 6 / 6 | 0 / 6 |
| Unrelated questions | 4 / 4 | 4 / 4 |
| The same facts asked again after one sleep | 6 / 6 | |
| Decode, three answers of 266 to 400 tokens | 119.8 tokens/s | 122.4 tokens/s |
| First token of those three short prompts | 1.03 s | 0.15 s |
| KV cache with the profile's defaults | 669,427 tokens | 766,241 tokens |
| Sleep | one cycle, 87.8 s, finished | |
| Tracebacks in the log | 0 |
The memory-off column is the base engine's, measured with the V14 image's memory off on the same checkpoint. The model card has the details.
vllm-v15-patch/; the build stops if the base image's files differ from those
originals.Content type
Image
Digest
sha256:460d8b68b…
Size
12 GB
Last updated
3 days ago
docker pull lovewin10/vllm-v15:release-10152026-v15