HUB
Sign inSign up

lovewin10/sglang-v15

By lovewin10

•Updated 3 days ago

Image
0

228

lovewin10/sglang-v15 repository overview

⁠sglang-v15

Modified lmsysorg/sglang:nightly-dev-cu13-20260924-ffac53d7⁠ by lovewin10. Not an official SGLang image.

The SGLang inside the base image is replaced by a newer SGLang source tree (upstream commit 1be97a0a63, version 0.5.24.dev) that carries the V15 episodic memory: models whose directory names a V15 architecture are served with a memory that persists across requests. A fact told in one chat is answered in a later, separate chat, and while the server is idle the memory sleeps: the served model itself replays what was stored, merges what was told twice, retires a fact a later one corrects and bridges notes about the same person, thing or place. Nothing is trained and nothing is deleted. CUDA graphs, the radix/prefix cache and EAGLE speculative decoding (MTP draft) stay on; the entrypoint and the sglang serve command are the base image's, so a launch command written for the base image runs unchanged apart from a writable /memory mount.

Against lovewin10/sglang-v14⁠: V14's world model is gone (small predictors fitted to the store, the only training V14 did, and the event they expected next, which decoding attended), and the sleep is new. The memory serves V15 model directories (any other checkpoint is served as the base image serves it); a store written by the V14 image is not read.

⁠What is changed

  • SGLang source. /sgl-workspace/sglang is the newer tree, installed as the base image installs it (editable), with its dependencies: torch 2.14.1+cu130, flashinfer 0.7.3, sglang-kernel 0.4.10, transformers 5.19.0.

  • V15 architectures. A V15 model directory keeps its backbone's checkpoint and adds memory options and remote code for Transformers. In SGLang its architecture is mapped to the native model when the configuration is loaded, so every engine feature of the native model applies. No model subclass, no architecture list is changed.

    V15 architecturenative SGLang modelmemory in this image
    Qwen3_5V15ForConditionalGeneration, Qwen3_5V15ForCausalLMQwen3.5 denseyes, tested on Qwen3.8-27B (FP8 weights)
    Qwen3_5MoeV15ForConditionalGeneration, Qwen3_5MoeV15ForCausalLMQwen3.5 MoEyes, same code as dense; not run on real weights
    NemotronHV15ForConditionalGeneration, NemotronHV15ForCausalLMNemotron-Hyes, same code as V14's tested one; not run with this image (text requests only; requests with images, video or audio run without memory)
    GptOssV15ForCausalLMGPT-OSSloads with SGLANG_V15_MEMORY=0 only
    Qwen4ExpV15ForConditionalGenerationQwen4-Exploads with SGLANG_V15_MEMORY=0 only (the memory for this model is in lovewin10/vllm-v15⁠)

    LFM3 V15 directories cannot be served: this SGLang has no LFM3 model.

  • The memory (sglang/srt/v15). The keys and values the attention layers compute are cut into events and stored with, on hybrid models, a low-rank engram of the state each event leaves in the recurrent layers (Gated DeltaNet, Mamba2). A request's prompt is read once to see what it asks for; the stored events whose keys would take its attention are recalled (one decision for every layer and head), their engrams are composed into the request's recurrent state, and the prompt is read again with the recalled keys and values merged into the attention by log-sum-exp. While decoding the merge runs inside the CUDA graphs over static per-request buffers (a prefill's merge has its own kernel); the Triton kernels are checked against torch ops at start-up and torch ops serve if they differ. Events are written after prefill on a background thread; generated tokens are not written. Every decision is the model directory's own code (the core the Transformers class runs), called by the manager.

  • The sleep (sglang/srt/v15/sleep.py). A thread of the server process waits until no user request has come for SGLANG_V15_SLEEP_AFTER seconds, then runs one cycle of the core's sleep through the server's own HTTP API: a replay is a /generate request read with the memory read-only (it writes nothing, its vote is captured), a question to the model is a /v1/chat/completions request with the memory off and thinking off. The requests carry ids beginning V15_DREAM_ and cache salts of their own, so they share no prefix cache with user requests and are not counted as activity. A cycle replays the newest events, takes a note out of recall when the model says twice that it only asks something, merges an event told again, asks the model in two wordings whether a later note corrects an earlier one it calls up (two yeses retire the earlier one: it is recalled only together with the later one, before it), dreams a question for some events and reads it, asks the model whether pairs of notes (from the dreams, from what a replay calls up together, from a shared rare word) which person, thing or place both name, and bridges the pairs where the name it gives is written in both notes (a bridged event needs 35% of the best vote instead of half), and scales down the bridges no prompt or dream used. Its edits go through the manager's writer thread; a user's request stops the cycle at its next step.

  • Engine hooks (14 files of SGLang, 136 added lines and 10 removed; the backbones' algorithms are untouched): the FlashInfer backend can return the attention log-sum-exp, the model runner calls the memory around forwards and may run a prefill again, the recurrent layers hand over per-event states, the scheduler registers and releases requests (and tells the memory when it is idle), warm-up and health-check requests write nothing.

⁠Models

Converted model directories on the Hugging Face Hub (the original weights, with the memory options and the memory code added); each card has its launch command and what was measured:

⁠Run

docker run --rm -it --gpus all --ipc=host -p 8000:8000 `
  -v "<folder containing the model>:/model:ro" `
  -v "<folder for the memory store>:/memory" `
  lovewin10/sglang-v15:nightly-dev-cu13-20260924-ffac53d7-v15 `
  sglang serve --model-path /model/<V15 model directory> --trust-remote-code `
  --host 0.0.0.0 --port 8000 --mem-fraction-static 0.78 --served-model-name EVE-27B `
  --chat-template /model/<V15 model directory>/chat_template.jinja `
  --reasoning-parser qwen3-thinking --tool-call-parser qwen3_coder `
  --attention-backend flashinfer --linear-attn-backend triton --mamba-ssm-dtype float32 `
  --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 `
  --enable-linear-replayssm-spec --mamba-radix-cache-strategy extra_buffer_lazy `
  --context-length 262144 --max-prefill-tokens 32768 --max-running-requests 8 `
  --kv-cache-dtype fp8_e4m3 --max-mamba-cache-size 16

For Nemotron-H use the arguments of SGLang's cookbook page for the checkpoint (they differ from the Qwen ones above); only the image, the model directory and the /memory mount change.

Environment variables:

  • SGLANG_V15_MEMORY=0 serves the model without memory, as the plain checkpoint.
  • SGLANG_V15_MEMORY_HOME is the store directory (default /memory); one sub-directory per set of weights.
  • SGLANG_V15_SLEEP_AFTER is how many seconds without a user request start a sleep (default 600); SGLANG_V15_SLEEP=0 turns the sleep off. SGLANG_V15_SELF_URL is the address the sleep sends its requests to (default: the server's own, 127.0.0.1 when it listens on every interface); with --api-key the key is sent along.
  • SGLANG_V15_MAX_BUFFER_GB bounds the static decode buffers (default 24).
  • SGLANG_V15_PROFILE=1 logs the time the memory spends in its eager sections (slows the server).
  • SGLANG_V15_TRACE=1 logs every recall decision: the best votes, the events recalled, the events composed into the recurrent states.
  • SGLANG_V15_INDEX_GB bounds the GPU memory the index keeps bounds and pooled keys in (default 2, allocated as the store grows). With more payloads than that a vote reads the rest back from the store (seconds a vote from a hard disk) and the log says so once.
  • SGLANG_V15_WRITE_QUEUE_GB bounds the RAM of events waiting for the background writer (default 8); a request whose events would exceed it stores nothing more, and the log says so.
  • Every 10 minutes the log has a line V15 memory: store ... events; index holds ... rows ...; process RSS ..., and every sleep cycle a line V15 sleep: cycle N finished after S s: replayed ..., merged ..., superseded ....

⁠Measured

One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-16, with this image as it is, the command above, -e SGLANG_V15_SLEEP_AFTER=20 (so that the checks see a sleep) and an empty store; the model is the 27B's FP8 directory, beside it the same directory with SGLANG_V15_MEMORY=0. A handful of questions per check: a functional check, not a benchmark. Every statement and every question is a chat of its own.

Qwen3.8-27B FP8 (Qwen3_5V15ForConditionalGeneration, the serve arguments above)

CheckMemory onMemory off
Ready (weights read from a hard disk)358 s439 s
KV cache at --mem-fraction-static 0.781,179,418 tokens1,179,418 tokens
Facts told in one chat, asked in another6 / 60 / 3
Unrelated questions4 / 44 / 4
The same after 40 other chats6 / 6, 4 / 4
Four chats telling and asking at once3 / 4
A fact in a 9,442-token prompt; a picture's colour; a stream dropped by the clientrecalled; recalled; server keeps answering
Corrected facts, before and after a sleep6 / 6, then 6 / 6 (no old value)
Two-step questions, before and after a sleep3 / 6, then 3 / 6
First token, three short prompts (mean)0.68 s0.21 s
Decode, three replies of up to 400 tokens (mean)76.0 tokens/s83.0 tokens/s
Sleep cycles (64 events each)3 finished, 87 to 95 s each, none interrupted
Prefill forwards in a CUDA graph; decode forwards in one1133 / 1141; 122 / 12219 / 19; 27 / 27
Tracebacks in the log00

On a store holding only the corrected and two-step chats, with the published core (the run above asked the bridge questions in the order it found the pairs; the published core asks those that share the rarest words first): corrected facts 5 / 6 (one old value) before one sleep, 6 / 6 after it; two-step questions 4 / 6 before, 5 / 6 after. That sleep (62 s) retired the six old values and made nine bridges, each between two notes that name the same person, thing or place. The first question that recalls something after a restart takes some 3.5 s longer than later ones.

⁠Limits

  • Two-step questions are the weak case, and a corrected fact is retired only by a sleep: until the server has slept the old and the new note are both in the store. The model cards give each model's figures.
  • The sleep uses the GPU while the server is idle: a cycle replays up to 64 events (asking, of each one with a question mark, whether it only asks), judges up to 128 pairs (two questions each), dreams up to 16 questions and asks up to 64 pairs which name they share. A user's request is served beside the sleep's request in flight and stops the cycle at its next step.
  • No separation between users. Every request reads and writes the same store, which keeps what the prompts contained (token ids, attention keys and values). Run one server with its own /memory for each user or tenant whose memories must stay apart; delete the directory to forget.
  • Engine settings. The memory needs an attention backend that returns the log-sum-exp at its layers: flashinfer (the default), fa3, triton, or Qwen4-Exp's sparse backend; a model with learned attention sinks needs triton. trtllm_mha returns none on this card (its XQA kernel), so a configuration in which SGLang picks it for decoding is refused with the reason. --enable-torch-compile works with the memory (verified), and so do the fp8 and fp4 (fp4_mx_block16, with triton) KV caches. TP, PP and DP attention of 1, no context parallelism and no PD disaggregation: these are refused at start-up, with the reason.
  • Pictures, videos and audio are stored and recalled as whole events (Nemotron's video as the pretrained model reads it, pruned by EVS); a picture the scheduler only has as a transported tensor is named by SGLang's own identity of it. A prompt given as embeddings is stored as a prompt of pictures.
  • The message after a system message begins an event of its own, so a system prompt that every conversation repeats is stored once and never recalled into another conversation.

⁠Source

The changes to SGLang are in /sgl-workspace/sglang inside the image (python/sglang/srt/v15 and the hooks, each marked). SGLang is Apache-2.0; this image is not affiliated with the SGLang project.

Tag summary

Content type

Image

Digest

sha256:7f149dead…

Size

19.5 GB

Last updated

3 days ago

docker pull lovewin10/sglang-v15:nightly-dev-cu13-20260924-ffac53d7-v15