Continuous Batching

Continuous batching is one of the inference first principles [[YuKaichao|游凯超]] says practitioners need in 148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子”. The episode treats it as part of the basic systems knowledge required to judge whether a large-model serving optimization is real.

In the wiki’s terms, continuous batching belongs between High-Throughput Inference Batching and product-facing AI Inference Cost Structure. It is not only a GPU-utilization trick: request arrival patterns, sequence lengths, attention state, and user latency targets all affect whether batching improves actual serving economics.

Key Claims

• Efficient inference requires scheduling active requests over time, not only choosing a model and counting tokens.
• Batching has to be evaluated with latency, memory pressure, attention-state management, and changing sequence lengths.
• [[VLLM|vLLM]] is important partly because inference engines make these scheduling choices reusable for many users and models.
• Agent workloads complicate batching because long tool loops, variable prompts, and cached prefixes create irregular request shapes.

Connections

• High-Throughput Inference Batching, AI Inference Cost Structure, and Inference Acceleration Stack — serving-efficiency context.
• [[VLLM|vLLM]], PagedAttention, and Prefix Caching — vLLM and state-management context.
• Agent Harness, Test-Time Scaling, and Model-Infra Co-Design — agent and co-design context.