Attention Residues

Attention Residues are described in 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ as [[KimiK3|Kimi K3]]’s mechanism for improving information flow across model depth. The source says ordinary residual connections add shallow-layer outputs into deeper layers, but as models get deeper, newly written information can be diluted by accumulated residual streams.

The source’s interpretation is that Attention Residues rotate attention from the sequence direction into the layer direction. Instead of every deeper layer receiving a simple sum of earlier representations, it can selectively read shallower-layer information, which may preserve useful features more flexibly.

Key Claims

• Attention Residues address depth-wise information flow, not only long-context sequence flow.
• The mechanism is compared with other multi-stream or compressed-residual approaches but is described as more attention-like and selective.
• Its upside is higher expressive capacity; its practical value still depends on implementation and training stability.
• In the episode’s broader frame, Attention Residues are one reason “Transformer” now covers a family of heavily modified architectures.

Connections

• Kimi K3, Kimi Delta Attention / KDA, and NoPE / No Position Encoding — K3 architecture branch.
• Transformer Architecture, Frontier Model Scaling, and Model-Infra Co-Design — model-design context.
• Zeng Zhiyuan / 曾志远 and Zhao Chenyang / 赵晨阳 — guests explaining the architecture.
• Mixture of Experts, Quantile Balancing, and Per-Head Muon — adjacent scaling and stability mechanisms.