NoPE / No Position Encoding

NoPE / no position encoding is discussed in 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ as part of [[KimiK3|Kimi K3]]’s long-context architecture. Zeng Zhiyuan / 曾志远 says K3 mostly removes explicit positional encoding and relies on [[KimiDeltaAttention|KDA]] recurrent state, gating, and decay to carry order and recency information implicitly.

The source says NoPE is not new to K3; Kimi Linear had already used it. The significance is that K3 scales the broader design to a much larger model while combining it with hybrid attention, Attention Residues, and progressive context extension toward million-token contexts.

Key Claims

• NoPE removes or minimizes explicit position embeddings rather than treating absolute or rotary position as the main order signal.
• KDA’s recurrent state and decay can encode sequence order and recency implicitly.
• NoPE’s value is tied to the surrounding architecture; it should not be evaluated as an isolated trick.
• Long-context capability still depends on memory capacity, global attention, training curriculum, and serving infrastructure.

Connections

• Kimi K3, Kimi Linear, Kimi Delta Attention / KDA, and Attention Residues — source architecture context.
• Agent Inference Workload, AI Inference Cost Structure, and Long-Horizon AI — long-context and agent-workload context.
• Model-Infra Co-Design and Inference Acceleration Stack — runtime implications.