Capability-Driven Robot Data Design / 能力反推机器人数据

Definition

Capability-driven robot data design selects data modalities, trajectories, and collection methods by reasoning backward from the behavior a model must learn rather than treating raw hours or one collection device as the objective.

Current Synthesis

The bounded source turns robot-data strategy into a capability-and-stage allocation problem. Smooth action needs smooth trajectories; recovery needs failures plus corrections; perception tasks such as segmentation, tracking, and depth reconstruction need different signals from new-task adaptation. Human video offers broad and inexpensive scene coverage, UMI-style data narrows the gap toward manipulation, simulation cheaply varies appearance and geometry, and real-robot teleoperation most closely matches the target body. The practical question is which mixture makes a specified capability learnable at pretraining, mid-training, or post-training time.

Key Claims

• The desired capability should determine data content, modality, and collection method.
• Failure recovery cannot be learned from success-only trajectories; failures and corrections must be represented.
• Human, UMI-style, simulation, and real-robot data occupy different points in cost, coverage, embodiment match, and controllability.
• Simulation is especially useful when controllable variation in position, texture, and color matters.
• Real-robot teleoperation is closest to deployment and therefore especially useful for fine-tuning or post-training.
• Morphology transfer is a necessary bridge when inexpensive human data is used to train robot action.

Evidence

• Capability-first evidence: “我看到了 Scaling Law 的信号”｜对谈清华叉院助理教授徐梦迪：具身智能、世界模型、真正的泛化 says smoothness, recovery, segmentation, tracking, reconstruction, and in-context learning imply different data requirements.
• Stage-allocation evidence: “我看到了 Scaling Law 的信号”｜对谈清华叉院助理教授徐梦迪：具身智能、世界模型、真正的泛化 assigns simulation to pretraining or mid-training and real-robot teleoperation to fine-tuning or post-training.
• Coverage evidence: “我看到了 Scaling Law 的信号”｜对谈清华叉院助理教授徐梦迪：具身智能、世界模型、真正的泛化 describes human data as cheap and broad, with UMI positioned between human and robot data.

Counterevidence & Qualifications

The source offers a design framework rather than measured allocation ratios. Morphology transfer remains an open technical problem, and broad human data may not contain the sensor, contact, latency, or action distributions of a target robot. The best mixture can change with body, task, model architecture, and deployment environment.

What Changed

• Created the concept to make desired model capability, rather than raw data volume, the organizing unit of robot-data strategy.

Related Concepts

• Embodied Robot Data Tradeoff - broader tradeoff among simulation, real machines, human data, and task-specific data.
• Real Robot Data Strategy - target-body grounding and physical feedback branch.
• UMI-Style Glove Data Collection / 手套式数据采集 - intermediate manipulation-data collection route.
• Robot Experience Data - robot-owned attempts and corrections needed for improvement and recovery.
• Sim2Real - transfer boundary between controlled synthetic variation and physical execution.
• Robot Data Scale Up - scale challenge qualified by capability relevance and data mixture.