Model Collapse

Model collapse is the failure mode in Kate Crawford: Mapping Empires where models trained repeatedly on synthetic outputs lose diversity, flatten minority patterns, erase outliers, and degrade toward lower-quality or noisier distributions. Kate Crawford also links the idea to model autophagy, where AI systems effectively consume their own outputs.

The concept turns synthetic data from a simple scaling solution into a risk that must be evaluated. It connects Frontier Model Scaling to Data Recipe Co-Creation: more data is not automatically better if the data is recursively generated, homogeneous, hallucinated, or detached from the human and physical variation the model needs to preserve.

Key Claims

• Repeated training on generated outputs can narrow the distribution a model learns.
• Minority patterns, rare cases, edge cases, and unusual styles are especially exposed when synthetic averages dominate.
• Synthetic data may still be useful, but it needs grounding, filtering, evaluation, and diversity controls.
• Media-scale AI Slop can become a data-quality problem if it enters future crawls.
• Model collapse matters beyond aesthetics because high-stakes systems may depend on rare or outlier cases.

Connections

• Kate Crawford - source speaker.
• AI Slop - synthetic media supply that can contaminate future training data.
• Frontier Model Scaling - broader scaling debate around data quantity and quality.
• Data Recipe Co-Creation - need to discover which data mixtures improve systems.
• AI Recognition Bias - related problem where model confidence can hide skewed or sparse training data.
• Human Judgment Under AI - review and evaluation remain necessary when generated outputs look plausible.