Record and Replay

Definition

Record and replay is a computer-use pattern in which a human demonstrates a GUI workflow and an agent converts the trace into a repeatable skill, reducing the need to rediscover every action from scratch.

Current Synthesis

The first source describes an OpenAI mechanism that records human computer work and turns it into repeatable skills, analogous to teleoperation data for robots. Vol. 175 adds direct practitioner experience: demonstrating a web task and asking Codex to repeat it was faster than making the model reason independently at every step. Together they support a hybrid view: demonstrations can transfer tacit workflow knowledge and reduce search cost, but the resulting routine still needs interface-change detection, permissions, validation, and recovery.

Key Claims

• Human demonstrations can transfer tacit GUI sequences into agent-readable routines.
• Reusing a demonstrated path can be faster and cheaper than open-ended visual reasoning at every step.
• Record and replay is more structured than free-form clicking but less flexible than a fully general computer-use agent.
• Repeatability depends on stable interfaces, bounded tasks, credential handling, and recovery from changed screens.
• Consequential replay still requires permission checks, verification, stopping rules, and rollback.

Evidence

• Mechanism and constraints: 171: 【AI季报 26Q2】从 coding 到 RSI，强者愈强的未来？ describes recording a human GUI workflow into a skill while flagging accuracy, latency, privacy, and permissions.
• Practitioner corroboration: Vol. 175 GPT 6 Astra、Opus 5.5、Jev 诸模型混战 reports that demonstrating a web workflow before Codex repeated it was more efficient than step-by-step autonomous judgment.

Counterevidence & Qualifications

Neither source provides controlled success rates, cost comparisons, interface-change tests, or evidence across high-stakes domains. A recorded path can fail silently when page structure, account state, localization, or permissions change. Demonstration reduces planning burden but does not establish that the intended action is safe or still appropriate.

What Changed

• Added hands-on corroboration that demonstration-first computer use can reduce repeated reasoning and improve efficiency.
• Migrated the page to the synthesis-first schema while preserving its complete evidence inventory.

Related Concepts

• Computer Use Agent - broader category for agents operating graphical interfaces.
• Agent Trajectory Distillation - learning relationship between recorded behavior and reusable action patterns.
• Agent Harness - execution layer for validation, retries, and recovery.
• Agent Permission Boundaries - authority constraints for replayed actions.
• AI Coding Verification - verification relationship when demonstrations affect code or production systems.