jtabet/sglang repository overview

⁠KVarN: KV Cache Compression for SGLang

KVarN adds KV cache compression to SGLang, reducing GPU memory usage for long-context inference.

⁠What It Does

• Int4 quantization of KV cache with dequantization on demand
• Hadamard rotation to improve quantization quality
• Sinkhorn equalization for balanced channel distributions
• Sliding window support for efficient eviction
• Speculative decode verify path
• CUDA graph compatible decode with flush during replay

⁠Available Tags

⁠Base Image

Built on top of lmsysorg/sglang:v0.5.12.post1-cu130 (CUDA 13.0, Ubuntu 24.04).

⁠Usage

docker pull jtabet/sglang:kvarn-latest
docker run --gpus all --rm   -p 30000:30000   jtabet/sglang:kvarn-latest   python3 -m sglang.launch_server     --model-path <model>     --kvarn     --kvarn-preset k4v2     --port 30000

⁠Source

• GitHub: jtabet/sglang⁠ (feature/kvarn branch)
• Upstream: sgl-project/sglang⁠