On-Policy Distillation

On-policy distillation is described in 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ as a post-training method where the student model generates its own trajectories and a teacher model scores them. The teacher therefore supplies denser reward signals over behavior the student actually produced, rather than only providing fixed offline answers.

The source contrasts this with off-policy distillation, where a teacher first generates a data set and the student imitates it offline. In K3’s discussion, on-policy distillation is tied to [[MOPDPostTraining|MOPD]], reasoning-effort variation, and the open question of whether a model can improve itself without an external scalable supervision signal.

Key Claims

• On-policy distillation keeps the training signal closer to the student’s current behavior distribution.
• Teacher scoring can supply denser feedback than sparse final-answer rewards.
• The method still depends on external supervision quality; it is not proof that a model can “distill itself” into unbounded improvement.
• Off-policy and on-policy distillation should be separated when evaluating model-training claims or copying accusations.

Connections

• Model Distillation / 模型蒸馏, MOPD Post-Training, Agent Post-Training, and Agent RL — training-method context.
• Kimi K3, Zeng Zhiyuan / 曾志远, and AI Verification — source and verifier context.
• Recursive Self-Improvement and Model Post-Training Bottleneck — boundary around self-improvement claims.