Attention Residues

151. 17岁被2026年ICML收录论文的小少年：我bet开心！开心！开心！ adds a comparison boundary through 苏廷昊. He distinguishes his attention projection residual work from Kimi K3’s Attention Residues and treats Kimi’s version as more strongly validated at industrial scale.

Attention Residues are described in 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ as Kimi K3’s mechanism for improving information flow across model depth. The source says ordinary residual connections add shallow-layer outputs into deeper layers, but as models get deeper, newly written information can be diluted by accumulated residual streams.

The source’s interpretation is that Attention Residues rotate attention from the sequence direction into the layer direction. Instead of every deeper layer receiving a simple sum of earlier representations, it can selectively read shallower-layer information, which may preserve useful features more flexibly.

Key Claims

• Attention Residues address depth-wise information flow, not only long-context sequence flow.
• The mechanism is compared with other multi-stream or compressed-residual approaches but is described as more attention-like and selective.
• Its upside is higher expressive capacity; its practical value still depends on implementation and training stability.
• In the episode’s broader frame, Attention Residues are one reason “Transformer” now covers a family of heavily modified architectures.
• Episode 151 adds that related residual ideas can appear in independent student research, but scale and validation level need to be kept distinct from Kimi K3’s source-described system.

Connections

• Kimi K3, Kimi Delta Attention / KDA, and NoPE / No Position Encoding — K3 architecture branch.
• Transformer Architecture, Frontier Model Scaling, and Model-Infra Co-Design — model-design context.
• Zeng Zhiyuan / 曾志远 and Zhao Chenyang / 赵晨阳 — guests explaining the architecture.
• Mixture of Experts, Quantile Balancing, and Per-Head Muon — adjacent scaling and stability mechanisms.
• Su Tinghao / 苏廷昊, Attention Projection Residuals, and AI-Native Youth Research — student-research comparison branch.