Structured 3D Robot Data

Structured 3D robot data is the episode’s answer to why internet image and video scale is not enough for manipulation. In E244｜端到端vs上下分层：机器人路径之争，正在转向？, Han Zheng / 韩正 argues that robots need accurate geometry, material, friction, elasticity, parts, and dynamics, especially when tasks require millimeter-level contact.

The source traces a data lineage from ImageNet to ShapeNet, PartNet / PartNet Mobility, and PartNet Mobility, then argues that the remaining gap is still large. A robot opening, grasping, inserting, or screwing objects needs structured physical affordances, not only a mesh, label, or video prior.

Key Claims

• 2D images and video can imply 3D structure but usually do not provide precise contact-relevant geometry and dynamics.
• Open-world manipulation needs data on object parts, motion constraints, material properties, friction, deformation, and task affordances.
• Non-structured 3D asset collections can be large while still containing duplicates, game assets, or objects that are poor training material for robots.
• Structured 3D data is valuable because it feeds Sim2Real, Robotics Simulation Evaluation, and low-level manipulation skill learning.

Connections

• Su Hao / 苏浩, ImageNet, ShapeNet, and PartNet / PartNet Mobility - research and data lineage.
• Open-World Robot Manipulation - capability target.
• Embodied Data Pyramid and Real Robot Data Strategy - adjacent data strategy concepts.
• 速度科技 / Sudu Technology - company building around this source’s route.