Test-Time Scaling

Test-time scaling is the pattern of spending more computation during inference to improve output quality. In 148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子”, [[YuKaichao|游凯超]] separates it into several workload shapes: repeated sampling and aggregation, longer thinking-token generation, and multi-turn [[AgentHarness|agent]] interaction with tools and environments.

The distinction matters because each form stresses infrastructure differently. Repeated sampling stresses throughput, long thinking stresses generation length and memory, while agent loops stress prompt stability, Prefix Caching, tool-call latency, and irregular request patterns.

Key Claims

• Test-time scaling should not be treated as one generic inference workload.
• More inference compute can improve quality, but it also changes cost, latency, batching, cache behavior, and system design.
• Agentic forms of test-time scaling make the harness part of the inference problem.
• Model teams, inference-engine teams, and product teams need shared vocabulary before optimizing these workloads.

Connections

• Agent Harness, Prefix Caching, and Model Harness Co-Evolution — agent and harness side.
• AI Inference Cost Structure, Continuous Batching, and [[VLLM|vLLM]] — serving cost and scheduling context.
• Model-Infra Co-Design and Open Source AI Infrastructure — broader system design context.