Model Distillation / 模型蒸馏

从蒸馏到合成数据到 RSI，模型竞争的下一个焦点是什么？｜对谈 Evolvent AI 联创孟繁青 adds [[MengFanqing|孟繁青]]’s stronger qualification: distillation is useful, especially when a weaker model has large learning margin, but it is not the decisive reason domestic models have become strong. The source says Chinese model progress should also be attributed to pretraining, architecture and efficiency work, data, and organization.

177: 详解Kimi K3：强到冲击Anthropic估值的模型什么样？ adds a post-training workflow version through [[KimiK3|Kimi K3]]. Zeng Zhiyuan / 曾志远 distinguishes classic distillation, off-policy distillation, and On-Policy Distillation, then uses [[MOPDPostTraining|MOPD]] to show distillation as capability merging across domain experts and reasoning-effort levels, not only small-model compression or copying.

Model distillation is the machine-learning technique discussed in E246｜何谓蒸馏？聊聊硅谷如何看中国开放模型逼近前沿, where a smaller or cheaper model learns behavior from a stronger teacher model. [[WangTiezhen|王铁镇]] distinguishes the classic form, where a student can learn from output probability distributions or logits, from the public accusation that a model was built by scraping another model’s text outputs.

The concept matters because the episode treats “distillation” as several different claims that should not be collapsed. A model can use teacher outputs legally or openly, violate an API provider’s terms by mass generation, learn from already-public model outputs on the internet, or be falsely accused because its behavior resembles another model. The source argues that proving [[KimiK3|Kimi K3]]’s core capability came from a specific closed model would require evidence beyond model identity confusion or short timing narratives.

Key Claims

• Distillation is a standard technique, not inherently a misconduct category.
• Closed APIs usually do not expose the logits needed for classic full-probability distillation.
• Training on generated text is a weaker and broader claim than classic distillation.
• DeepSeek is used as an example of a model family whose R1-era outputs and artifacts were described as more open to downstream distillation.
• Distillation can improve smaller models, but the source says architecture, data engineering, RL, inference optimization, and Scaling Efficiency also have to be considered.
• Terms-of-service violations, copying accusations, and technical distillation are different questions.
• In agent post-training, distillation may be used to merge domain specialists into one model, but it still needs reliable teacher scoring and external supervision.
• In environment-based Synthetic Agent Data, using an outside model such as Claude to explore and save traces may be called distillation, but the environment, task, verifier, and scoring system can be the harder part of the data product.
• Distillation can be a shortcut that risks weakening a team’s own insight if it becomes a substitute for understanding the main training path.

Connections

• Synthetic Agent Data, Environment-Based Agent Benchmarks, RSI Data, and Evolvent AI — Evolvent AI source branch where distillation overlaps with synthetic trajectory generation.
• Kimi K3, Moonshot AI / 月之暗面, DeepSeek, and Open Source AI Models - model cases in the episode.
• Model Identity Data Pollution / 模型身份数据污染 - why self-identification errors are weak evidence for distillation.
• Scaling Efficiency, Model Post-Training Bottleneck, and Agent Post-Training - adjacent model-improvement mechanisms.
• Open Model Safety Governance and AI Governance And Compliance - compliance and transparency layer when model-generated data is used.
• On-Policy Distillation, MOPD Post-Training, Kimi K3, and AI Verification - K3 post-training branch added by LateTalk episode 177.