HUB
Sign inSign up

lovewin10/vllm-v15

By lovewin10

•Updated 3 days ago

Image
0

207

lovewin10/vllm-v15 repository overview

⁠vllm-v15

Modified ghcr.io/local-inference-lab/vllm:release-10152026⁠ by lovewin10. Not an official vLLM, FlashInfer or local-inference-lab image.

The local-inference-lab build of vLLM (V2 model runner, b12x kernels in FlashInfer) with the V15 episodic memory added: a model directory that names the V15 architecture is served with a memory that persists across requests, so a fact told in one chat is answered in a later, separate chat, and while the server is idle the memory sleeps: the served model itself replays the stored events, merges repeated ones, retires the ones a later event corrects and bridges related ones (no training, no new content). It is made for LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15⁠, whose FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales) this vLLM build serves and Transformers cannot load. CUDA graphs (FULL_AND_PIECEWISE), the prefix cache, MTP speculative decoding and async scheduling stay on; the entrypoint and the PROFILE mechanism are the base image's.

⁠What is changed

  • vllm/v15 (new): the memory manager and its sleep (the same code as the SGLang port's, with Triton kernels for the decode-step and prefill merges, checked against torch ops at start-up), the vLLM side of it (the sleep reaches the model through the server's own OpenAI API), and a Transformers compatibility layer for the V15 configuration.
  • Five files of the base image, each marked as modified:
    • vllm/transformers_utils/config.py: the V15 architecture name is mapped to the native Qwen4ExpForConditionalGeneration, so every list, profile and the MTP drafter treat it as the native model; with the memory on, the b12x QSA plans are asked for their log-sum-exp.
    • vllm/v1/worker/gpu/model_runner.py (V2 runner): the manager is built once the weights are loaded (before the memory profile) and its sleep started, requests are registered and released, and the memory is called around each forward.
    • vllm/models/qwen4_exp/nvidia/b12x_qsa.py: recalled events are merged into each b12x QSA call by log-sum-exp, before the output gate.
    • vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py: after the b12x GDN prefill, the zero-start state of each stored event (its engram).
    • flashinfer/experimental/b12x/attention/qsa/_contract.py: with B12X_QSA_RETURN_LSE=1, plans without context parallelism return the log-sum-exp too.

⁠Run

On one 96 GB GPU (an RTX PRO 6000 Blackwell), with the model repository downloaded to a folder:

docker run -d --name flash-next-v15 --gpus all --ipc=host --shm-size 32g \
  --ulimit memlock=-1 --ulimit stack=67108864 -p 127.0.0.1:8000:8000 \
  -e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 \
  -v <model folder>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory \
  lovewin10/vllm-v15:release-10152026-v15
  • VLLM_V15_MEMORY=0: serve the folder as the plain checkpoint, exactly as the base image does.
  • VLLM_V15_SLEEP=0: the memory without its sleep; VLLM_V15_SLEEP_AFTER (default 600): seconds without a user request before a sleep cycle starts. The sleep calls the server at http://127.0.0.1:$PORT (VLLM_V15_SELF_URL to change it; an API key as VLLM_API_KEY). Every cycle logs a V15 sleep: line.
  • VLLM_V15_MAX_SLOTS (default: --max-num-seqs, 16 with this profile): requests that read the memory at once, so every request the engine runs has its memory; about 100 MiB each, taken from the KV cache. A smaller number saves that memory; requests beyond it then run without memory.
  • VLLM_V15_INDEX_GB (unset by default): the store's index rows are reserved at start-up for the payloads it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB); a number pins the size.
  • VLLM_V15_WORK_GB (default 1.25): GPU memory held back from the KV cache for the memory's work the vLLM profile does not see (a prefill's recurrent states, recent keys, far contexts).
  • VLLM_V15_KV_CACHE_GB (default 0.25): GPU memory for recently recalled events' decoded keys.
  • VLLM_V15_TRACE=1: log every recall decision.

The GPU shows nearly full with or without the memory: the profile gives everything below --gpu-memory-utilization 0.96 to the KV cache. A lower -e GPU_MEMORY_UTILIZATION (or a fixed -e KV_CACHE_MEMORY_BYTES=..., the base image's options) leaves GPU memory free, all of it taken from the KV cache: about 0.96 GiB, some 52,000 tokens, for every 0.01. Below about 0.90 one 262,144-token request no longer fits and vLLM stops after loading the weights, so add -e MAX_MODEL_LEN=-1 as well (vLLM serves the longest context that fits and logs it as Auto-fit max_model_len) or a smaller MAX_MODEL_LEN.

The store lives in /memory; delete the folder to forget. Every request reads and writes the same store: run one server with its own /memory for each user whose memories must stay apart. A V14 store is not read: V15 opens a store of its own.

⁠Measured

One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, this image as it is with LTM-Lab/Qwen3.8-Flash-Next-FP4-Local-Abliterated-v15⁠, the profile above, an empty store and -e VLLM_V15_SLEEP_AFTER=20; every statement and every question a chat of its own, temperature 0, thinking off. A handful of questions per check, not a benchmark.

Memory onMemory off (the base engine)
A fact told in one chat, asked in a new chat6 / 60 / 6
Unrelated questions4 / 44 / 4
The same facts asked again after one sleep6 / 6
Decode, three answers of 266 to 400 tokens119.8 tokens/s122.4 tokens/s
First token of those three short prompts1.03 s0.15 s
KV cache with the profile's defaults669,427 tokens766,241 tokens
Sleepone cycle, 87.8 s, finished
Tracebacks in the log0

The memory-off column is the base engine's, measured with the V14 image's memory off on the same checkpoint. The model card has the details.

⁠Source and license

  • The patch that builds this image (Dockerfile, the changed files, the originals they were made from) is in the model repository under vllm-v15-patch/; the build stops if the base image's files differ from those originals.
  • vLLM and FlashInfer are Apache-2.0, and so are the patch and the memory code. Everything else in the image is the base image's, under its components' own licenses (the base is built on NVIDIA's PyTorch container).

Tag summary

Content type

Image

Digest

sha256:460d8b68b…

Size

12 GB

Last updated

3 days ago

docker pull lovewin10/vllm-v15:release-10152026-v15