Model-Infra Co-Design
E247｜对话盛颖：xAI，Infra的浪漫，SGLang，开源，平权与“甄嬛传” adds the SGLang and [[RadixARC|Redix ARK]] version through [[ShengYing|盛颖]]. The source makes co-design operational: Radix Attention, Prefix Caching, Day-Zero Model Support, inference/RL overlap, and new-model architecture churn all require the serving engine and model behavior to be understood together.
177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ adds a [[KimiK3|Kimi K3]] technical-report version. K3’s [[KimiDeltaAttention|KDA]], Flash KDA kernels, Per-Head Muon, Quantile Balancing, QAT, Kernel Development Agents, and AgentIn all show that model architecture, optimizer, kernel lifecycle, serving stack, chip support, and agent environment have to be designed together.
Model-infra co-design is [[YuKaichao|游凯超]]’s core technical frame in 148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子”. The episode uses an electricity analogy: hardware is the natural resource, the model is the generator, and the inference engine is the power system that determines whether tokens can be produced efficiently.
The concept says model architecture, serving engine, hardware, and agent workload should be designed together. After easy general-purpose compute gains slow down, efficiency depends on whether attention design, cache behavior, quantization, routing, batching, chip features, and [[AgentHarness|agent harnesses]] line up.
Key Claims
- Model teams and infrastructure teams cannot optimize independently once inference cost, latency, memory, and power become first-order constraints.
- Hardware Lottery makes some algorithms more durable than others because hardware support determines whether they can be run efficiently.
- [[VLLM|vLLM]] is a co-design layer because inference engines translate model design into deployable serving behavior.
- [[MixtureOfExperts|MoE]], long context, test-time compute, and coding-agent loops create different serving bottlenecks.
- Good co-design begins at model-design time, not after a trained model is thrown over the wall to an infrastructure team.
- Agent environments and training-time quantization also belong in co-design when the model is optimized for long-running tool use, not just chat serving.
- SGLang adds that day-zero compatibility and prefix-cache data structures make co-design a production-maintenance problem, not only a model-design ideal.
Connections
- SGLang, Radix Attention, Day-Zero Model Support, [[RadixARC|Redix ARK]], and AI Infrastructure As Product - source-247 inference-engine branch.
- [[VLLM|vLLM]], PagedAttention, Continuous Batching, and Prefix Caching — inference-engine examples.
- AI Inference Cost Structure, Inference Acceleration Stack, and High-Throughput Inference Batching — serving economics and runtime optimization.
- AI Chip Specialization, AI Infrastructure Full-Stack Moat, and Hardware Lottery — hardware and platform fit.
- DeepSeek, Open Source AI Models, and Open Source AI Infrastructure — open-model and infrastructure ecosystem case.
- Agent Harness, Test-Time Scaling, and Model Harness Co-Evolution — workload and harness side of co-design.
- Kimi K3, Kimi Delta Attention / KDA, Per-Head Muon, Kernel Development Agents, AgentIn, and Moore Threads / 摩尔线程 - K3 architecture, kernel, and chip-adaptation branch added by LateTalk episode 177.