Multi-Teacher Distillation

Definition

Multi-teacher distillation is a post-training pattern where a student model learns from several teacher models or specialized expert models instead of a single teacher.

Current Synthesis

In the Kimi K3 discussion, multi-teacher distillation is less a small-model compression trick than a capability-integration workflow. Different teachers can encode different reward functions, reasoning-effort levels, domains, or agent skills; the student then becomes the merge target for those partial capabilities. The wiki treats this as adjacent to MOPD and on-policy distillation: it can reduce coordination complexity across post-training teams, but it does not remove the need for high-quality teachers, evaluation, provenance control, and task-specific reward design.

Key Claims

• Multiple teachers can represent domain-specific capabilities or reward preferences that would be hard to optimize in one shared RL run.
• The method can be organizational: separate teams can build expert models, then merge capability into a student through distillation.
• Multi-teacher setups can improve beyond one teacher on narrow tasks if teachers cover complementary strengths.
• The approach can also import teacher mistakes, refusal patterns, style artifacts, or provenance risk from several sources at once.

Evidence

• K3 post-training workflow: 177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ describes MOPD as a way to merge domain expert models and reasoning-effort levels into one K3 post-trained model.
• Project-management rationale: 152. 领读Kimi K3技术报告：从架构创新聊起，注意力美学、多教师蒸馏和开源MoE says multi-teacher distillation can simplify post-training management by treating different RL rewards and specialized abilities as teacher models to be merged.
• Boundary condition: 152. 领读Kimi K3技术报告：从架构创新聊起，注意力美学、多教师蒸馏和开源MoE still ties the method to OPD, QAT, MTP, and inference design, showing that teacher count alone is not the full recipe.

Counterevidence & Qualifications

Multi-teacher distillation does not prove autonomous self-improvement or clean provenance. The value depends on teacher quality, task coverage, filtering, evaluation, and whether the student can resolve conflicts among teachers rather than averaging incompatible behavior. It also inherits the governance questions attached to Model Distillation / 模型蒸馏 when teacher outputs come from restricted or opaque systems.

What Changed

• Adds an explicit concept for the multi-teacher part of K3’s post-training discussion.
• Separates capability merging from generic model distillation and from on-policy teacher scoring.

Related Concepts

• Model Distillation / 模型蒸馏 - parent technique and governance debate.
• On-Policy Distillation - related method where teacher feedback follows student-produced behavior.
• MOPD Post-Training - K3-adjacent workflow for merging specialized expert models.
• Agent Post-Training - training stage where multiple environments, rewards, and skills may need to be combined.
• Kimi K3 - central model case in the source.