Attention Projection Residuals

Attention projection residuals are the source-scoped model-architecture idea 苏廷昊 describes in 151. 17岁被2026年ICML收录论文的小少年：我bet开心！开心！开心！. Starting from value residual learning, he says he added normalization on residual paths, extended the residual idea from value to key and query projections, and widened the first-layer attention projection so one split serves the current layer while another split carries information forward.

The source distinguishes this from Attention Residues in Kimi K3. Su treats his approach as related attention-depth information flow, but he is cautious about claiming more than his own scale demonstrates. The page should therefore be read as an episode-grounded research note rather than a full technical report.

Key Claims

• The mechanism tries to preserve early-layer attention projection information across later layers.
• The source’s described change is projection-specific: value, key, and query pathways matter, not only the ordinary residual stream.
• The proposed first-layer width split creates a separation between current-layer computation and residual information reserved for later layers.
• The evidence level is source-scoped because the podcast summary does not include paper tables, ablations, or reproduction details.
• The idea connects to the broader shift where Transformer variants modify attention, residuals, cache behavior, and training stability while still remaining Transformer-adjacent.

Connections

• Su Tinghao / 苏廷昊 and ICML 2026 — source researcher and publication venue.
• Transformer Architecture, Attention Residues, and Kimi Delta Attention / KDA — adjacent attention-architecture branch.
• AI Research Feedback Compression and Research Taste — experiment loop and judgment context.
• AI-Native Youth Research — youth-research pathway that produced the source-scoped idea.