vLLM

vLLM is the open-source large-model inference engine at the center of 148. 对游凯超3小时访谈：开源Infra、和模型Co-design 、“如果vLLM失败，我们会后悔一辈子”. [[YuKaichao|游凯超]] traces it from the PagedAttention paper and Berkeley open-source culture into a production infrastructure project supported by Infract and governed through the [[PyTorchFoundation|PyTorch Foundation]].

The episode frames vLLM’s moat less as one clever trick than as the accumulated work of maintaining compatibility with rapidly changing model structures, attention variants, hardware constraints, and user workloads. Its 2024 V0-to-V1 rewrite is presented as a response to growing system complexity while preserving user-facing stability where possible.

For the wiki, vLLM is a concrete case of Open Source AI Infrastructure. It sits inside AI Inference Cost Structure because serving tokens cheaply and reliably depends on scheduling, cache management, model support, hardware fit, and community maintenance. It also sits inside Model-Infra Co-Design because inference engines increasingly influence how model teams think about attention, [[MixtureOfExperts|MoE]], context length, and deployment efficiency.

Key Claims

• vLLM began from PagedAttention but became valuable as a full inference engine and community-maintained system.
• The project was donated to the [[PyTorchFoundation|PyTorch Foundation]] to keep the trademark and governance community-owned.
• Infract supplies company-level labor, customer collaboration, and infrastructure resources around the open project.
• vLLM must support many model architectures while also deleting features that no longer fit mainstream inference workloads.
• Coding agents create more low-quality pull requests, increasing the importance of maintainer judgment and real user feedback.
• Its long-term ambition in the source is to become Linux-like infrastructure for AI inference.

Connections

• PagedAttention, Continuous Batching, and Prefix Caching — state and scheduling concepts around efficient inference.
• AI Inference Cost Structure, High-Throughput Inference Batching, and Inference Acceleration Stack — serving economics and optimization context.
• Model-Infra Co-Design, Hardware Lottery, and AI Chip Specialization — model/hardware/system fit.
• DeepSeek, Kimi, and Open Source AI Models — model ecosystem that creates pressure on engine support.
• [[PyTorchFoundation|PyTorch Foundation]], Infract, and Open Source AI Infrastructure — governance and sustainability context.