Self-Supervised Learning

Self-supervised learning is discussed in 133. 对谢赛宁的7小时马拉松访谈：世界模型、逃出硅谷、AMI Labs、两次拒绝Ilya、杨立昆、李飞飞和42 as one route for Representation Learning. Xie Saining describes its movement from pretext tasks into contrastive learning and MoCo-style work at FAIR with Kaiming He.

Source View

The source treats self-supervised learning as important but incomplete. Xie says it produced strong results yet did not deliver the same scalable future paradigm that GPT-style systems appeared to deliver. He also argues that language models are not pure self-supervised learning in the usual sense because language already embeds human interpretation, abstraction, and labels created by civilization.

Connections

• Representation Learning — broader objective self-supervision serves.
• Kaiming He, Xie Saining, and FAIR — people and lab context for MoCo-style work.
• Diffusion Transformers and Joint Embedding Predictive Architecture — later predictive and generative architecture directions.
• World Models and Multimodal Intelligence — domains where self-supervision is insufficient unless paired with better abstractions and world grounding.
• Frontier Model Scaling — source tension between scalable objectives and data/model form.