Model Distillation / 模型蒸馏

Definition

Model distillation is the transfer of behavior, reasoning patterns, outputs, or task trajectories from a stronger teacher model or set of models into a student model.

Current Synthesis

The wiki’s current view is that distillation is a standard technical family, not a misconduct label by itself. The bounded sources distinguish classic logits-style compression, generated-output fine-tuning, multi-teacher capability transfer, agent-trajectory imitation, and public accusations that a model copied a closed provider. Those are different claims with different evidence, legal, and strategic requirements.

The newest source sharpens the small-model side of the concept. For 逯雨鑫 / 逯雨昕, teacher-generated or teacher-shaped data can help a narrow student model through SFT, but only when it moves the model distribution toward the intended task. The result can be a good distillation even if unrelated benchmarks fall, while capacity gap means a smaller model may not absorb the teacher’s full behavior.

Key Claims

Distillation is a legitimate model-training technique, but it becomes contentious when source provenance, terms of service, or competitive model improvement are disputed.
Closed APIs usually do not expose the full probability distributions associated with classic distillation, so many public disputes concern generated text or behavior traces instead.
Good distillation is target-aligned behavior transfer, not merely collecting fluent teacher answers.
Agent-era distillation can include full task trajectories, tools, environments, and feedback rather than static question-answer pairs.
Distillation can narrow gaps for weaker or smaller models, but architecture, pretraining, data engineering, RL, inference optimization, and organization can still be decisive.
Stronger evidence for improper distillation requires behavior distributions, refusal patterns, code style, account traces, traffic evidence, or other provenance signals beyond self-identification errors.
Capacity gap and benchmark tradeoffs limit what small student models can learn from stronger teachers.

Evidence

Technical and evidence boundary

E246｜何谓蒸馏？聊聊硅谷如何看中国开放模型逼近前沿 distinguishes classic distillation from public copying accusations and rejects model self-identification as enough evidence.
179: 蒸馏风暴：一场无人公开谈论的技术竞赛 expands the evidence standard to behavior-distribution analysis, refusal patterns, code style, call traces, account evidence, and anti-distillation enforcement.

Governance and organization

中国消费者带动拉夫劳伦增长，东航优化机票退改签政策 records the governance-first case where 张一鸣 reportedly opposed distillation because U.S.-model provenance disputes could harm TikTok and weaken team development.
179: 蒸馏风暴：一场无人公开谈论的技术竞赛 adds that terms-of-service, legal, and organizational-learning risks can dominate pure technical speed.

Model-factory and agent data

177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ describes On-Policy Distillation and MOPD as post-training mechanisms for combining domain expert models, reasoning effort levels, and teacher scoring.
从蒸馏到合成数据到 RSI，模型竞争的下一个焦点是什么？｜对谈 Evolvent AI 联创孟繁青 links distillation to Synthetic Agent Data while stressing that environments, task design, and scoring can be harder than copying teacher answers.

Small-model target alignment

一个人、两周、数百美元，如何训出登顶 Hugging Face 的模型 | 对谈研究员逯雨鑫 treats SFT as practical distillation when teacher outputs improve a narrow student model in the target domain, while warning that other benchmarks can drop and capacity gap can block full transfer.

Counterevidence & Qualifications

The sources repeatedly reject distillation as a single explanation for Chinese model progress; architecture, data, efficiency, RL, inference, and organization remain part of the causal picture.
A public accusation of improper distillation is not proven by timing, similarity, or a model claiming to be another model.
Distillation can import teacher mistakes, refusal patterns, or style artifacts if the data pipeline is weak.
For narrow application use, a benchmark drop outside the target may be acceptable; for general-purpose releases, the same tradeoff may be unacceptable.

What Changed

Migrated the page to synthesis-v1 and compressed prior source-led material into claim groups.
Added the small-model practitioner view: good distillation is target alignment under capacity constraints, not universal improvement.
Tightened the evidence distinction between technical distillation, ToS risk, and public accusation.

Supervised Fine-Tuning / SFT - method that can carry teacher behavior into a student model.
Data-First Post-Training / 数据优先后训 - data-quality discipline that determines whether distillation improves the target behavior.
Agent Trajectory Distillation - agent-era form based on task traces rather than static answers.
Synthetic Agent Data - data source that can overlap with distillation when a stronger model generates trajectories.
Model Post-Training Bottleneck - broader bottleneck around data, evaluation, and capacity.
AI Model Distillation Governance - compliance and organizational boundary around using teacher outputs.
Model Distillation Evidence - evidence standard for claims that distillation occurred.

Sources

6 source notes across 4 shows