Modified lmsysorg/sglang:nightly-dev-cu13-20260924-ffac53d7 by
lovewin10. Not an official SGLang image.
The SGLang inside the base image is replaced by a newer SGLang source tree (upstream commit 1be97a0a63,
version 0.5.24.dev) that carries the V15 episodic memory: models whose directory names a V15
architecture are served with a memory that persists across requests. A fact told in one chat is answered
in a later, separate chat, and while the server is idle the memory sleeps: the served model itself
replays what was stored, merges what was told twice, retires a fact a later one corrects and bridges notes
about the same person, thing or place. Nothing is trained and nothing is deleted. CUDA graphs, the
radix/prefix cache and EAGLE speculative decoding (MTP draft) stay on; the entrypoint and the
sglang serve command are the base image's, so a launch command written for the base image runs
unchanged apart from a writable /memory mount.
Against lovewin10/sglang-v14: V14's world model is gone
(small predictors fitted to the store, the only training V14 did, and the event they expected next, which
decoding attended), and the sleep is new. The memory serves V15 model directories (any other checkpoint is
served as the base image serves it); a store written by the V14 image is not read.
SGLang source. /sgl-workspace/sglang is the newer tree, installed as the base image installs it
(editable), with its dependencies: torch 2.14.1+cu130, flashinfer 0.7.3, sglang-kernel 0.4.10,
transformers 5.19.0.
V15 architectures. A V15 model directory keeps its backbone's checkpoint and adds memory options and remote code for Transformers. In SGLang its architecture is mapped to the native model when the configuration is loaded, so every engine feature of the native model applies. No model subclass, no architecture list is changed.
| V15 architecture | native SGLang model | memory in this image |
|---|---|---|
Qwen3_5V15ForConditionalGeneration, Qwen3_5V15ForCausalLM | Qwen3.5 dense | yes, tested on Qwen3.8-27B (FP8 weights) |
Qwen3_5MoeV15ForConditionalGeneration, Qwen3_5MoeV15ForCausalLM | Qwen3.5 MoE | yes, same code as dense; not run on real weights |
NemotronHV15ForConditionalGeneration, NemotronHV15ForCausalLM | Nemotron-H | yes, same code as V14's tested one; not run with this image (text requests only; requests with images, video or audio run without memory) |
GptOssV15ForCausalLM | GPT-OSS | loads with SGLANG_V15_MEMORY=0 only |
Qwen4ExpV15ForConditionalGeneration | Qwen4-Exp | loads with SGLANG_V15_MEMORY=0 only (the memory for this model is in lovewin10/vllm-v15) |
LFM3 V15 directories cannot be served: this SGLang has no LFM3 model.
The memory (sglang/srt/v15). The keys and values the attention layers compute are cut into events
and stored with, on hybrid models, a low-rank engram of the state each event leaves in the recurrent
layers (Gated DeltaNet, Mamba2). A request's prompt is read once to see what it asks for; the stored
events whose keys would take its attention are recalled (one decision for every layer and head), their
engrams are composed into the request's recurrent state, and the prompt is read again with the recalled
keys and values merged into the attention by log-sum-exp. While decoding the merge runs inside the CUDA
graphs over static per-request buffers (a prefill's merge has its own kernel); the Triton kernels are
checked against torch ops at start-up and torch ops serve if they differ. Events are written after
prefill on a background thread; generated tokens are not written. Every decision is the model
directory's own code (the core the Transformers class runs), called by the manager.
The sleep (sglang/srt/v15/sleep.py). A thread of the server process waits until no user request
has come for SGLANG_V15_SLEEP_AFTER seconds, then runs one cycle of the core's sleep through the
server's own HTTP API: a replay is a /generate request read with the memory read-only (it writes
nothing, its vote is captured), a question to the model is a /v1/chat/completions request with the
memory off and thinking off. The requests carry ids beginning V15_DREAM_ and cache salts of their own,
so they share no prefix cache with user requests and are not counted as activity. A cycle replays the
newest events, takes a note out of recall when the model says twice that it only asks something,
merges an event told again, asks the model in two wordings whether a later note corrects an earlier one
it calls up (two yeses retire the earlier one: it is recalled only together with the later one, before
it), dreams a question for some events and reads it, asks the model whether pairs of notes (from
the dreams, from what a replay calls up together, from a shared rare word) which person, thing or
place both name, and bridges the pairs where the name it gives is written in both notes (a bridged
event needs 35% of the best vote instead of half), and scales down the bridges no prompt or dream used. Its edits go through the manager's writer
thread; a user's request stops the cycle at its next step.
Engine hooks (14 files of SGLang, 136 added lines and 10 removed; the backbones' algorithms are untouched): the FlashInfer backend can return the attention log-sum-exp, the model runner calls the memory around forwards and may run a prefill again, the recurrent layers hand over per-event states, the scheduler registers and releases requests (and tells the memory when it is idle), warm-up and health-check requests write nothing.
Converted model directories on the Hugging Face Hub (the original weights, with the memory options and the memory code added); each card has its launch command and what was measured:
docker run --rm -it --gpus all --ipc=host -p 8000:8000 `
-v "<folder containing the model>:/model:ro" `
-v "<folder for the memory store>:/memory" `
lovewin10/sglang-v15:nightly-dev-cu13-20260924-ffac53d7-v15 `
sglang serve --model-path /model/<V15 model directory> --trust-remote-code `
--host 0.0.0.0 --port 8000 --mem-fraction-static 0.78 --served-model-name EVE-27B `
--chat-template /model/<V15 model directory>/chat_template.jinja `
--reasoning-parser qwen3-thinking --tool-call-parser qwen3_coder `
--attention-backend flashinfer --linear-attn-backend triton --mamba-ssm-dtype float32 `
--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 `
--enable-linear-replayssm-spec --mamba-radix-cache-strategy extra_buffer_lazy `
--context-length 262144 --max-prefill-tokens 32768 --max-running-requests 8 `
--kv-cache-dtype fp8_e4m3 --max-mamba-cache-size 16
For Nemotron-H use the arguments of SGLang's cookbook page for the checkpoint (they differ from the Qwen ones
above); only the image, the model directory and the /memory mount change.
Environment variables:
SGLANG_V15_MEMORY=0 serves the model without memory, as the plain checkpoint.SGLANG_V15_MEMORY_HOME is the store directory (default /memory); one sub-directory per set of weights.SGLANG_V15_SLEEP_AFTER is how many seconds without a user request start a sleep (default 600);
SGLANG_V15_SLEEP=0 turns the sleep off. SGLANG_V15_SELF_URL is the address the sleep sends its
requests to (default: the server's own, 127.0.0.1 when it listens on every interface); with --api-key
the key is sent along.SGLANG_V15_MAX_BUFFER_GB bounds the static decode buffers (default 24).SGLANG_V15_PROFILE=1 logs the time the memory spends in its eager sections (slows the server).SGLANG_V15_TRACE=1 logs every recall decision: the best votes, the events recalled, the events composed
into the recurrent states.SGLANG_V15_INDEX_GB bounds the GPU memory the index keeps bounds and pooled keys in (default 2,
allocated as the store grows). With more payloads than that a vote reads the rest back from the store
(seconds a vote from a hard disk) and the log says so once.SGLANG_V15_WRITE_QUEUE_GB bounds the RAM of events waiting for the background writer (default 8); a
request whose events would exceed it stores nothing more, and the log says so.V15 memory: store ... events; index holds ... rows ...; process RSS ...,
and every sleep cycle a line V15 sleep: cycle N finished after S s: replayed ..., merged ..., superseded ....One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, with this image as it is, the
command above, -e SGLANG_V15_SLEEP_AFTER=20 (so that the checks see a sleep) and an empty store; the
model is the 27B's FP8 directory, beside it the same directory with SGLANG_V15_MEMORY=0. A handful of
questions per check: a functional check, not a benchmark. Every statement and every question is a chat of
its own.
Qwen3.8-27B FP8 (Qwen3_5V15ForConditionalGeneration, the serve arguments above)
| Check | Memory on | Memory off |
|---|---|---|
| Ready (weights read from a hard disk) | 358 s | 439 s |
KV cache at --mem-fraction-static 0.78 | 1,179,418 tokens | 1,179,418 tokens |
| Facts told in one chat, asked in another | 6 / 6 | 0 / 3 |
| Unrelated questions | 4 / 4 | 4 / 4 |
| The same after 40 other chats | 6 / 6, 4 / 4 | |
| Four chats telling and asking at once | 3 / 4 | |
| A fact in a 9,442-token prompt; a picture's colour; a stream dropped by the client | recalled; recalled; server keeps answering | |
| Corrected facts, before and after a sleep | 6 / 6, then 6 / 6 (no old value) | |
| Two-step questions, before and after a sleep | 3 / 6, then 3 / 6 | |
| First token, three short prompts (mean) | 0.68 s | 0.21 s |
| Decode, three replies of up to 400 tokens (mean) | 76.0 tokens/s | 83.0 tokens/s |
| Sleep cycles (64 events each) | 3 finished, 87 to 95 s each, none interrupted | |
| Prefill forwards in a CUDA graph; decode forwards in one | 1133 / 1141; 122 / 122 | 19 / 19; 27 / 27 |
| Tracebacks in the log | 0 | 0 |
On a store holding only the corrected and two-step chats, with the published core (the run above asked the bridge questions in the order it found the pairs; the published core asks those that share the rarest words first): corrected facts 5 / 6 (one old value) before one sleep, 6 / 6 after it; two-step questions 4 / 6 before, 5 / 6 after. That sleep (62 s) retired the six old values and made nine bridges, each between two notes that name the same person, thing or place. The first question that recalls something after a restart takes some 3.5 s longer than later ones.
/memory for each
user or tenant whose memories must stay apart; delete the directory to forget.flashinfer (the default), fa3, triton, or Qwen4-Exp's sparse backend; a model with learned attention
sinks needs triton. trtllm_mha returns none on this card (its XQA kernel), so a configuration in which
SGLang picks it for decoding is refused with the reason. --enable-torch-compile works with the memory
(verified), and so do the fp8 and fp4 (fp4_mx_block16, with triton) KV caches. TP, PP and DP attention
of 1, no context parallelism and no PD disaggregation: these are refused at start-up, with the reason.The changes to SGLang are in /sgl-workspace/sglang inside the image (python/sglang/srt/v15 and the
hooks, each marked). SGLang is Apache-2.0; this image is not affiliated with the SGLang project.
Content type
Image
Digest
sha256:7f149dead…
Size
19.5 GB
Last updated
3 days ago
docker pull lovewin10/sglang-v15:nightly-dev-cu13-20260924-ffac53d7-v15