Embodied Data Pyramid

Embodied data pyramid is 谢晨’s frame for combining data sources in Embodied AI. In 134. 【数据的综述】和谢晨聊，新时代的石油、历史、版图、数据金字塔、定价与Recipe, the top layer is real robot teleoperation or body data, the middle layer is simulation data, and the bottom layer is internet-scale and human first-person data.

170: 【具身季报 26Q2】世界模型大风不停，和不想被贴标签的人 extends the top and bottom of this pyramid by naming concrete collection shifts: Aloha-style teleoperation, UMI-style body-free data, first-person video, whole-body motion capture, and dexterous-hand datasets. The source’s Embodied Robot Data Paradigms concept keeps the pyramid dynamic by asking which new data source unlocks which robot capability.

从会跳舞到有感知，触觉是机器人通往智能的门票吗？｜ S10E19 adds a tactile-data layer through Yimu Technology / 一目科技. Eric Li Zhiqiang / 李志强 says real tactile robot data is valuable but expensive, simulation data should be expanded with Optical Tactile Sensing, and large-scale video data can still help pretrain visual priors before touch and action are added.

E244｜端到端vs上下分层：机器人路径之争，正在转向？ adds a structured-3D manipulation layer. Han Zheng / 韩正 argues that real robot operation, teleoperation, motion capture, and first-person video are too scarce or incomplete to cover arbitrary object manipulation, so Structured 3D Robot Data and Sim2Real need to sit between internet-scale priors and real robot validation.

147. 和蚂蚁灵波沈宇军聊：机器人原生基础模型、大脑和本体的关系、预训练与数据scale up、老师汤晓鸥 adds a source that pushes harder toward real-machine data. 沈宇军 still accepts first-person human data and simulation for evaluation or partial coverage, but 蚂蚁灵波 treats real robot data, stricter cleaning, and many body configurations as the main near-term route for Robot Data Scale Up.

Key Claims

• Real robot data is most physically accurate but expensive, hard to scale, and therefore too narrow to carry general robotics learning alone.
• Robotics Simulation Evaluation is the scalable middle layer because it can generate repeated tasks, failures, counterfactuals, and evaluation scenarios.
• Human first-person and internet data are larger and less body-specific, but can provide scene, object, task, and daily-life priors.
• The pyramid should be a loop: real-world and human data can be converted into simulation worlds, while simulation outputs must be checked against real-world results.
• This creates a tension with pure Real Robot Data Strategy: real data stays necessary, but the source argues it should not dominate the recipe by default.
• Hardware-specific dexterous-hand data complicates the pyramid because “real robot data” does not transfer cleanly when hand structure, sensors, or drive method change.
• Tactile Sensing creates a special data problem because it is high-frequency and continuous while also being closer to force and deformation ground truth than ordinary visual data.
• TouchNet is proposed as a field-level tactile dataset, but the source still places it inside a broader recipe of scarce real data, simulation, and video pretraining.
• Structured 3D Robot Data adds a geometry, parts, material, friction, and dynamics layer that ordinary video and teleoperation cannot supply cleanly.
• The Ant Lingbo source creates a useful tension inside the pyramid: simulation and recipes may be necessary for scale, but real-machine data may remain the source of sensor, body, and execution distribution that robot-native models need most.

Connections

• Data As Education — broader metaphor behind the pyramid.
• Data Recipe Co-Creation — process for discovering how much of each layer improves a model.
• Physical World Data Flywheel — adjacent real-world data-loop concept from the Xinghaitu source.
• World Models and Vision Language Action Models — model routes that need physical-world data and evaluation.
• 光轮智能 — company building simulation-centered data infrastructure in the source.
• Embodied Robot Data Paradigms, Dexterous Manipulation, and Robot Teleoperation and Remote Takeover — collection-method updates from the LateTalk source.
• Tactile Sensing, Optical Tactile Sensing, TouchNet, and Tactile Transformer Encoder — tactile data, sensor route, dataset, and model-interface additions from the What’s Next source.
• 速度科技 / Sudu Technology, ShapeNet, PartNet / PartNet Mobility, and Sim2Real — structured 3D and simulation route added by E244.
• 蚂蚁灵波 / Ant Lingbo, 沈宇军 / Shen Yujun, Robot Data Scale Up, and Embodied Native Foundation Models — real-machine-heavy scale-up route added by episode 147.