Multimodal Intelligence

Multimodal intelligence is the source’s route from language-only systems toward visual, spatial, continuous, and eventually predictive world understanding. In 133. 对谢赛宁的7小时马拉松访谈：世界模型、逃出硅谷、AMI Labs、两次拒绝Ilya、杨立昆、李飞飞和42, Xie Saining describes stages from pure language models, to show-and-tell image QA, to continuous visual streams and spatial understanding.

Claire Isabel Webb & Nina Miolane: The Geometry of Consciousness adds a computational-neuroscience example of spatial intelligence. Nina Miolane shows that both biological navigation circuits and artificial networks trained on position prediction can organize around Neural Geometry such as the Spatial Navigation Torus, suggesting that spatial representation may have task-driven mathematical structure.

Key Claims

• Language is a powerful interface, but not the whole world and not the only form of thought or decision.
• Visual and spatial intelligence need representations that can process continuous perceptual streams, not just isolated images.
• Spatial intelligence can be studied through population-level geometry rather than only through input/output behavior.
• A final route should move toward predictive World Models that explain the observed world and support planning.
• Over-tokenizing visual streams for LLMs may hide the physical structure that models need to learn.

Connections

• Xie Saining, Fei-Fei Li, and ImageNet — people and dataset context from the computer-vision path.
• Representation Learning and Self-Supervised Learning — technical foundations.
• World Models, Video Models, and Diffusion Transformers — adjacent model routes.
• Language User Interface — language as interface, not complete substrate.
• Embodied AI and AI Plus Terminals — physical and sensor-rich deployment contexts.
• Nina Miolane, Neural Geometry, Spatial Navigation Torus, and Fourier Spatial Encoding — spatial-representation branch added by the Long Now source.