Podcast Atlas
concept Updated 2026-08-08 Tags: Ai, Agents, Inference, Infrastructure

Agent Inference Workload

E247｜对话盛颖：xAI，Infra的浪漫，SGLang，开源，平权与“甄嬛传” adds the SGLang implementation angle. [[ShengYing|盛颖]] explains Radix Attention as a way to reuse shared prefixes and KV cache in multi-turn and agent-like settings, making the agent workload a concrete serving-engine problem.

177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ adds a K3-specific version of the workload. Zhao Chenyang / 赵晨阳 says a coding task may reuse a very large stable prefix while adding a much smaller increment, making prefix reuse central to latency and cost. The complication is that [[KimiDeltaAttention|KDA]] carries mutable recurrent state, so prefix caching and speculative-sampling rollback are harder than with append-only KV cache.

Agent inference workload is the serving pattern described in E246｜何谓蒸馏？聊聊硅谷如何看中国开放模型逼近前沿 where agent systems differ from ordinary chatbots or simple RAG. [[WangTiezhen|王铁镇]] says agent work often has long input, short output, and high prefix/KV-cache reuse, making cache lifetime, scheduling, and hardware/software co-design central to cost.

The concept explains why open-model competition creates infrastructure opportunity. If many agent tasks reuse long context or tool state, the provider that manages Prefix Caching, routing, batching, memory, and model fit can reduce cost even when it does not own the strongest base model.

Key Claims

  • Agent workloads can be dominated by context reuse and intermediate steps rather than only final output length.
  • KV-cache lifecycle and prefix reuse become product and infrastructure variables.
  • Routing across models can matter more when agents decompose work into planning, tool use, retrieval, execution, and review.
  • Hardware/software co-design remains a major cost-reduction opportunity for agent-heavy inference.
  • Mutable attention state can make agent serving harder even when it reduces memory growth.
  • Radix Attention adds that tree-structured prefix reuse can be a first-class serving mechanism when many conversations or agent steps share context.

Connections