Robot Scaling Claim Caution / 幂律现象与 scaling law 谨慎

Definition

Robot scaling claim caution is the discipline of separating suggestive improvements in training or held-out loss from a validated law connecting more data and larger models to safe, reliable success on unseen physical tasks.

Current Synthesis

Both bounded sources report early scaling-like signals while refusing the strongest conclusion. The Shenpu interview calls its internal observation a power-law phenomenon rather than a scaling law because sufficient controlled experiments and public evaluation are missing. Xu Mengdi points to source-reported held-out-loss improvement as pretraining data grows from roughly one hundred thousand to one million hours, but adds the decisive robotics qualification: lower average loss may not improve task success when a small error at a contact-critical step can break a long-horizon action. A meaningful embodied scaling law would therefore connect model and data growth to sustained unseen-task success under disclosed task, data, safety, and latency conditions.

Key Claims

• Power-law-like improvement and held-out-loss reduction are signals, not by themselves validated robot scaling laws.
• The most meaningful outcome variable is success on unseen tasks, not average prediction loss alone.
• Contact-critical and long-horizon tasks can amplify small local errors into complete task failure.
• Controlled scaling experiments require known data quality, cost, usability, diversity, and data-to-model ratios.
• Large human or egocentric data collections can be cheap to record but expensive to annotate, train, and validate for robot use.
• Zero-shot results should disclose task difficulty and data exposure rather than stand alone as general-capability claims.

Evidence

• Claim-discipline evidence: 于是转身向具身走去｜对话王家伟：24 岁的具身智能首席科学家 says the team observes power-law phenomena but lacks the experiments needed to claim a scaling law.
• Controlled-data evidence: 于是转身向具身走去｜对话王家伟：24 岁的具身智能首席科学家 connects controlled parameter and data-ratio experiments to self-collected, cost-known UMI data reported as over 90% usable.
• Loss-signal evidence: “我看到了 Scaling Law 的信号”｜对谈清华叉院助理教授徐梦迪：具身智能、世界模型、真正的泛化 reports seeing lower unseen-test loss as some companies move from roughly one hundred thousand to one million hours of pretraining data.
• Success-rate qualification: “我看到了 Scaling Law 的信号”｜对谈清华叉院助理教授徐梦迪：具身智能、世界模型、真正的泛化 argues that critical contact errors can break a whole long-horizon task and defines the desired law through unseen-task success.
• Cost and disclosure evidence: 于是转身向具身走去｜对话王家伟：24 岁的具身智能首席科学家 distinguishes cheap raw egocentric footage from annotation and training cost and calls for disclosure around zero-shot and few-shot data use.

Counterevidence & Qualifications

Neither source supplies audited scaling curves, model sizes, complete data composition, confidence intervals, or reproducible cross-company tests. The million-hour comparison is reported as an observation from company releases rather than an independent experiment. The current evidence supports a research signal and a measurement standard, not the existence, exponent, or commercial implication of a robot scaling law.

What Changed

• Added held-out-loss improvement as a stronger early signal while preserving the distinction between signal and validated law.
• Made unseen-task success the required outcome and added the contact-critical reason loss can diverge from success.
• Broadened the validation boundary to include task disclosure, safety, and latency alongside data and model scale.

Related Concepts

• Robot Data Scale Up - broader data-scaling bottleneck this concept qualifies.
• Robot Generalization Performance Tradeoff - capability tension a useful scaling law must measure.
• Egocentric Robot Data - large-scale first-person route whose total cost and transfer value remain uncertain.
• Robot Evaluation Problem - benchmark and disclosure layer required to test the scaling claim.
• Capability-Driven Robot Data Design / 能力反推机器人数据 - reminder that capability-relevant data composition matters alongside hours.
• Frontier Model Scaling - language-model scaling discourse that cannot be imported into robotics without embodied outcome evidence.