Prefix Caching

E247｜对话盛颖：xAI，Infra的浪漫，SGLang，开源，平权与“甄嬛传” adds Radix Attention as the SGLang-specific prefix-caching mechanism. [[ShengYing|盛颖]] says a radix tree can track shared prefixes and reuse already computed KV cache, especially in multi-turn dialogue and agent workloads.

177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ adds the [[KimiK3|Kimi K3]] edge case. The source says coding-agent workloads can contain huge reusable prefixes, but [[KimiDeltaAttention|KDA]] makes reuse harder because recurrent state is updated and overwritten rather than simply appended. That turns prefix caching from a prompt-layout optimization into an architecture-specific serving problem.

Prefix caching is the inference reuse pattern highlighted in 148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子” through coding agents. [[YuKaichao|游凯超]] says coding-agent prompts often contain stable material such as system instructions, repository context, tool descriptions, and policies, making cache reuse valuable when the prefix stays stable.

The source’s important twist is that Agent Harness design can destroy the cache. Dynamic dates, changing tool lists, reordered instructions, or frequently rewritten system prompts can make repeated work look different to the inference engine, increasing AI Inference Cost Structure even when the user task is similar.

Key Claims

• Prefix caching connects prompt engineering to serving economics.
• Agent harnesses should treat stable context as an infrastructure asset, not only as model input text.
• Cache-breaking changes can raise cost and latency without improving model quality.
• The pattern becomes more important under Test-Time Scaling because long or repeated agent loops amplify any avoidable prompt recomputation.
• Hybrid attention models may need custom cache lifecycle logic instead of assuming every model state behaves like a standard KV cache.
• Radix-tree cache management shows that the data structure inside the serving engine can be part of the product’s cost and latency profile.

Connections

• SGLang, Radix Attention, [[ShengYing|盛颖 / Sheng Ying]], and AI Infrastructure As Product - source-247 radix-tree cache branch.
• Agent Harness, Model Harness Co-Evolution, and Agentic Workflow — harness and workflow context.
• [[VLLM|vLLM]], Continuous Batching, and AI Inference Cost Structure — inference-engine and cost context.
• Model-Infra Co-Design and Inference Acceleration Stack — broader co-design and acceleration context.
• Kimi K3, Kimi Delta Attention / KDA, and Agent Inference Workload — mutable-state serving case added by LateTalk episode 177.