Robot Evaluation Problem

Robot evaluation problem is the difficulty of comparing real-world robot systems that K emphasizes in 146. 对Physical Intelligence柯丽一鸣4小时访谈：Pi的开源模型研究，机器人的江湖、族谱与主角. Unlike language models, robots cannot be cleanly ranked by a single public leaderboard because evaluation has to happen on real or realistic machines in variable physical scenes.

The source lists many confounders: lighting, position, background, table height, object angle, hardware state, and task definition can all change results. That makes private company evals useful internally but hard to compare externally, and it makes public demos a weak proxy for frontier status.

Key Claims

• A task’s success definition matters as much as the model architecture when judging progress.
• Real-machine evaluation is expensive, slow, and hardware-dependent.
• Throughput can combine speed and quality, but only for tasks whose success and time windows are clearly defined.
• Company demos should be read as evidence of capability, not as a complete frontier ranking.
• The problem connects to simulation because scalable evaluation may need repeatable simulated or semi-simulated testbeds.

Connections

• Robotics Simulation Evaluation — scalable evaluation infrastructure that can complement real-machine tests.
• Robot Generalization Performance Tradeoff — tradeoff that evaluation is supposed to measure.
• Physical Intelligence Pi Model Series — Pi sequence and π0.6* throughput example.
• Open-World Robot Manipulation, Structured 3D Robot Data, and Sim2Real — adjacent route for testing unseen-object manipulation.
• Figure AI, Generalist, Skild AI, and Physical Intelligence — companies whose public claims and demos need evaluation discipline.