lovewin10/sglang-v14 repository overview

⁠sglang-v14

Modified lmsysorg/sglang:nightly-dev-cu13-20260924-ffac53d7⁠ by lovewin10. Not an official SGLang image.

The SGLang inside the base image is replaced by a newer SGLang source tree (upstream commit 1be97a0a63, version 0.5.24.dev) that carries the V14 episodic memory: models whose directory names a V14 architecture are served with a memory that persists across requests. A fact told in one chat is answered in a later, separate chat. CUDA graphs, the radix/prefix cache and EAGLE speculative decoding (MTP draft) stay on; the entrypoint and the sglang serve command are the base image's, so a launch command written for the base image runs unchanged apart from a writable /memory mount.

⁠What is changed

• SGLang source. /sgl-workspace/sglang is the newer tree, installed as the base image installs it (editable), with its dependencies: torch 2.14.1+cu130, flashinfer 0.7.3, sglang-kernel 0.4.10, transformers 5.19.0.

• V14 architectures. A V14 model directory keeps its backbone's checkpoint and adds memory options and remote code for Transformers. In SGLang its architecture is mapped to the native model when the configuration is loaded, so every engine feature of the native model applies. No model subclass, no architecture list is changed.

  V14 architecture	native SGLang model	memory in this image
  Qwen3_5V14ForConditionalGeneration, Qwen3_5V14ForCausalLM	Qwen3.5 dense	yes, tested on Qwen3.8-27B
  Qwen3_5MoeV14ForConditionalGeneration, Qwen3_5MoeV14ForCausalLM	Qwen3.5 MoE	yes, same code as dense; not run on real weights
  NemotronHV14ForConditionalGeneration, NemotronHV14ForCausalLM	Nemotron-H	yes, tested on Nemotron 3.5 Super VL 120B NVFP4 (text requests; requests with images, video or audio run without memory)
  GptOssV14ForCausalLM	GPT-OSS	loads with SGLANG_V14_MEMORY=0 only
  Qwen4ExpV14ForConditionalGeneration	Qwen4-Exp	loads with SGLANG_V14_MEMORY=0 only

  LFM3 V14 directories cannot be served: this SGLang has no LFM3 model.

• The memory (sglang/srt/v14). The keys and values the attention layers compute are cut into events and stored with, on hybrid models, a low-rank engram of the state each event leaves in the recurrent layers (Gated DeltaNet, Mamba2). A request's prompt is read once to see what it asks for; the stored events whose keys would take its attention are recalled (one decision for every layer and head), their engrams are composed into the request's recurrent state, and the prompt is read again with the recalled keys and values merged into the attention by log-sum-exp. While decoding the merge runs inside the CUDA graphs over static per-request buffers. Events are written after prefill on a background thread; generated tokens are not written.

• Engine hooks (15 files of SGLang, 133 added lines; the backbones' algorithms are untouched): the FlashInfer backend can return the attention log-sum-exp, the model runner calls the memory around forwards and may run a prefill again, the recurrent layers hand over per-event states, the scheduler registers and releases requests, warm-up and health-check requests write nothing.

⁠Models

Converted model directories on the Hugging Face Hub (the original weights, with the memory options and the memory code added); each card has its launch command and what was measured:

• LTM-Lab/Qwen3.8-27B-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-v14⁠
• LTM-Lab/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-v14⁠
• LTM-Lab/Qwen3.5-0.8B-v14⁠

⁠Run

docker run --rm -it --gpus all --ipc=host -p 8000:8000 `
  -v "<folder containing the model>:/model:ro" `
  -v "<folder for the memory store>:/memory" `
  lovewin10/sglang-v14:nightly-dev-cu13-20260924-ffac53d7-v14 `
  sglang serve --model-path /model/<V14 model directory> --trust-remote-code `
  --host 0.0.0.0 --port 8000 --mem-fraction-static 0.78 --served-model-name EVE-27B `
  --chat-template /model/<V14 model directory>/chat_template.jinja `
  --reasoning-parser qwen3-thinking --tool-call-parser qwen3_coder `
  --attention-backend flashinfer --linear-attn-backend triton --mamba-ssm-dtype float32 `
  --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 `
  --enable-linear-replayssm-spec --mamba-radix-cache-strategy extra_buffer_lazy `
  --context-length 262144 --max-prefill-tokens 32768 --max-running-requests 8 `
  --kv-cache-dtype fp8_e4m3 --max-mamba-cache-size 16

For Nemotron-H use the arguments of SGLang's cookbook page for the checkpoint (they differ from the Qwen ones above); only the image, the model directory and the /memory mount change.

A V14 model directory is made from a checkpoint with the upgrade scripts of the V14 project (examples/<architecture>_reinstatement_upgrade.py --link hardlink): the checkpoint's own config.json with the architecture name, an auto_map and the memory options added, two code files, and links to the weights.

Environment variables:

• SGLANG_V14_MEMORY=0 serves the model without memory, as the plain checkpoint.
• SGLANG_V14_MEMORY_HOME is the store directory (default /memory); one sub-directory per set of weights.
• SGLANG_V14_MAX_BUFFER_GB bounds the static decode buffers (default 24).
• SGLANG_V14_PROFILE=1 logs the time the memory spends in its eager sections (slows the server).
• SGLANG_V14_TRACE=1 logs every recall decision: the best votes, the events recalled, the events composed into the recurrent states.

⁠Measured

One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-11. Every figure comes from running this image as it is, each model starting on an empty store. The questions are few (six facts, four unrelated questions, four concurrent chats, six corrected facts, six two-step questions): a functional check, not a benchmark. Every statement and every question is a chat of its own. For comparison, the 27B without memory (SGLANG_V14_MEMORY=0) answers none of the cross-chat facts, gives its first token after 0.09 to 0.30 s and decodes the same three prompts at 50.2, 58.5 and 46.9 tokens/s.

Qwen3.8-27B (Qwen3_5V14ForConditionalGeneration, the serve arguments above)

Nemotron 3.5 Super VL 120B NVFP4 (NemotronHV14ForConditionalGeneration, the cookbook's single-GPU arguments with EAGLE)

Qwen3.5-0.8B V14 directory with SGLANG_V14_MEMORY=0 (no memory)

⁠Notes and limits

• The memory needs an attention backend that returns the log-sum-exp: flashinfer (as in the command above) or fa3; with another one the server stops at startup and says so. TP, PP and DP attention must be 1.
• Differences from the Transformers implementation of V14: events are fixed-length (no surprise boundaries); the whole new part of a prompt is read again when something is recalled, up to three readings while the recall keeps changing; no time headers; while decoding, recall is reconsidered after 32 accepted tokens and then at doubling intervals; the MTP draft runs without memory.
• Events recalled while the prompt is read enter both pathways (attention and recurrent state), the engram composed in front of the prompt. Events recalled later, while decoding, are attended and their engrams composed behind what the request has read: composed in front there, an event would sit before the text being generated and the model would answer it instead.
• Voting again once recalled events are in place (while decoding, and after a prompt is read again) is an option of the model directory (recall_while_decoding). The upgrade script turns it off for Nemotron-H: measured on Nemotron 3.5 Super VL, a stored document that began by telling the reader how to answer was recalled during every generation and the reply obeyed it, while the decisions taken on the prompt's first reading were right. Qwen directories keep it on.
• Weak spots seen in the runs above, on the 27B:
  • Two-step questions ("in which rack is the backup server" needs "the backup server is called Ganymede" and then "Ganymede is in rack 9"). The first fact was always recalled; the second got a vote of 2.2 to 2.6 where the bar was half of the first one's (2.6 to 2.7), and was missed in three of six.
  • Notes that merely look like the prompt are recalled with votes just above the floor: eight or nine requests in ten recalled something at the prompt. The unrelated questions were still answered right; such a reply starts later (the prompt is read again) and decodes slower. The recall thresholds (recall_floor, recall_ratio in the model directory's memory options) were set on Qwen3.5-0.8B; on the 27B no floor separates the best votes of unrelated questions (0.5 to 2.1) from the lowest vote of a right answer (2.06), so they were left alone.
  • The first token comes about 0.5 s later than without memory: a prompt that recalls something is read two or three times.
  • A corrected fact is not retired: the old note and the new one are both stored, and recalled events carry no time stamp here.
• On Nemotron-H only facts asked for directly are dependable. Its votes are coarser (eight attention layers of two key-value heads, no positions in the attention), and once recalled events are in place every vote of the same tokens rises (a look-alike question that got 0.19 at the first vote got 0.88 on the re-reading). Its directory therefore has recall_while_decoding off: the first reading of a prompt's tokens decides, with no vote while decoding or after a re-reading. Measured so: cross-chat facts 5 of 6 (the paraphrased one is missed), corrected facts 0 of 6, two-step questions 1 of 6, and the fact inside a 14.8k-token document not recalled (vote 0.18, floor 0.29). With the later votes on, corrected facts were 4 of 6 but replies answered another stored question or acknowledged a recalled statement as if it had just been said: better a miss than the wrong target.
• Disk: about 45 KB per prompt token for the 27B; about 2 MB of state engram per event (up to 128 tokens) for Nemotron-H. Events unused for seven days are rewritten in a 4-bit format.
• A prefix served from the radix cache carries what the earlier request recalled.
• Events are stored by a background thread. A request first waits for the writes queued before it: what the server read before the request arrived is what it may recall, and its conversation's earlier turns must be in the store to be continued. A short prompt is stored in a fraction of a second, a 9k-token document in about twelve (27B, Docker Desktop mount). After 120 s the request stops waiting: one whose prompt holds text still queued writes nothing itself, any other goes on as usual; a request that cannot get the store at all runs without memory. A slow store never stops the server.
• The store holds what the prompts contained (token ids, attention keys and values), and every request reads and writes the same store: there is no separation between users. Run one server with its own /memory for each user or tenant whose memories must stay apart; delete the directory to forget.
• Nemotron-H: SGLang itself runs the prefill of a multimodal model without a CUDA graph ("Breakable CUDA graph is incompatible with multimodal model"), with or without the memory; decode and verify replay graphs.
• Base image license and terms apply (SGLang: Apache-2.0; the NVIDIA container license shown at startup). The changed SGLang files are marked in the source tree under /sgl-workspace/sglang (git status).