Kimi Delta Attention / KDA

Definition

Kimi Delta Attention / KDA is the Kimi-family linear-attention mechanism used in Kimi K3 as part of a hybrid attention architecture that alternates recurrent linear-attention state with periodic global attention.

Current Synthesis

KDA’s current wiki role is to show that long-context efficiency is becoming an architecture-and-kernel problem. The earlier LateTalk source frames KDA as a way to reduce KV-cache and memory movement for long contexts while complicating prefix reuse, rollback, and serving implementation. The newer Zhang Xiaojun technical reading gives the mechanism a more explicit lineage: RetNet-like decay, DeltaNet capacity, Gated DeltaNet, then KDA’s channelized decay. KDA is therefore not only “linear attention at scale”; it is a constrained design where stronger recurrence, numerical range control, chunk/tile kernels, global-attention retention, and inference-engine state management have to work together.

Key Claims

• KDA lowers long-context memory pressure by compressing much of the sequence history into recurrent state rather than a conventional full KV cache.
• K3 remains hybrid: retained global attention is important because fixed or compressed recurrent state alone can lose detail.
• Channelized decay gives KDA more expressive control over how different channels remember or forget context.
• KDA’s kernel feasibility depends on numerical and tiling constraints, including decay bounds that keep computation inside practical low-precision ranges.
• Mutable recurrent state makes Prefix Caching, speculative decoding rollback, and serving-engine support harder than append-only KV-cache handling.
• KDA raises the value of Model-Infra Co-Design because the mathematical form, kernel, cache lifecycle, and inference stack cannot be optimized separately.

Evidence

• Hybrid long-context design: 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ describes KDA plus periodic global attention as K3’s long-context architecture and explicitly ties it to prefix caching and rollback challenges.
• Lineage and channelized decay: 152. 领读Kimi K3技术报告：从架构创新聊起，注意力美学、多教师蒸馏和开源MoE traces KDA through RetNet, DeltaNet, Gated DeltaNet, and channelized decay.
• Kernel co-design: 152. 领读Kimi K3技术报告：从架构创新聊起，注意力美学、多教师蒸馏和开源MoE emphasizes lower-bound decay, tile/chunk ranges, BF16 practicality, and context-parallel chunking as part of why KDA is implementable.

Counterevidence & Qualifications

KDA does not eliminate all long-context problems. The sources preserve a hybrid design because global attention still helps retrieve older detail, and they treat serving support as a real complication rather than a solved afterthought. KDA’s benefit is also hardware- and implementation-dependent: a stronger formula can be unattractive if kernels, prefix cache behavior, or speculative decoding rollback become too expensive.

What Changed

• Adds the RetNet-to-DeltaNet-to-Gated-DeltaNet lineage behind KDA.
• Makes channelized decay and kernel/numerical constraints part of the current KDA synthesis.
• Separates KDA’s memory advantage from a broader claim that linear attention alone solves long-context retrieval.

Related Concepts

• Kimi K3 - model case where KDA is scaled and discussed.
• Kimi Linear - predecessor context for Kimi’s linear-attention direction.
• NoPE / No Position Encoding - complementary long-context design that relies partly on recurrent state for order and recency.
• Prefix Caching - serving optimization complicated by mutable KDA state.
• Inference Acceleration Stack - runtime layer affected by KDA kernels and cache lifecycle.
• Model-Infra Co-Design - co-design frame that KDA exemplifies.
• Transformer Architecture - broader architecture family whose components KDA modifies.