Tsinghua open-sources AKV, a KV cache engineering server for LLM agents on SGLang

Researchers from Tsinghua University and CUHK have open-sourced AKV, a serving system for KV cache engineering in LLM agents built on SGLang. Instead of summarizing history, agents drop selected KV cache entries and reuse the rest, cutting average cost by 27.7% and raising average task scores by 1.7 points. The code is Apache-2.0 licensed.

MarkTechPost

Open source

Average cost falls 27.7% while average task score rises 1.7 points
GPT-OSS-120B on SWE-bench Verified: cost per task drops from 20.0 to 9.7 cents, score rises from 40.0 to 43.2
Three cache operations: Append, Drop and Repos with RoPE rotation and no hidden-state recomputation
Tested on 8 A800-80GB GPUs in BF16 with an OpenAI-compatible API

#Tsinghua#SGLang#AKV

Tsinghua open-sources AKV, a KV cache engineering server for LLM agents on SGLang

Google Cloud ships llm-d 1.0 with cache-aware routing for TPU and GPU fleets

Tensormesh launches platform to cut LLM inference costs via KV-cache reuse

Lightbits Inferra KV Cache Engine Claims 16x Session Density and 10M-Token Contexts

DeepSeek open-sources DeepInfer engine, minus cache and load balancing