Agent Trust Calibration / 智能体信任校准

Definition

Agent trust calibration / 智能体信任校准 is the process of matching delegated autonomy and transaction value to demonstrated agent capability, bounded risk, traceable execution, and credible remedies rather than assuming either perfect safety or blanket distrust.

Current Synthesis

The source separates capability trust from risk trust. Users need reason to believe an agent can complete a task, but greater capability also increases the harm of misunderstood or overbroad action. Trust should therefore expand in steps: low-risk tasks and small amounts, explicit limits, observable execution, successful history, strong recourse, and progressively broader delegation.

The goal is not belief that agents never fail. It is justified confidence that authority is narrow, failures are detectable, responsibility can be assigned, and losses can be remedied. This makes compensation and dispute handling part of product trust rather than a back-office exception.

Key Claims

• Capability trust and risk trust are distinct and must both be earned.
• Appropriate delegation grows with task evidence, scoped authority, observability, and recourse.
• A user’s maximum delegated amount is a practical proxy for confidence in the whole transaction system.
• Trust accumulates through reliable institutional experience and may develop more slowly than underlying technical capability.
• Overtrust is itself a failure when polished convenience hides uncertain ranking, weak controls, or unavailable remedies.

Evidence

• Two-part trust evidence: 外滩大会线下圆桌｜敢把钱包交给AI吗？聊聊Agent交易爆发前夜的信任基建 records Pete Lau’s distinction between whether an agent can do the task and whether its autonomy remains controllable.
• Historical evidence: 外滩大会线下圆桌｜敢把钱包交给AI吗？聊聊Agent交易爆发前夜的信任基建 compares agent payment with the years required for consumers to trust ecommerce payment and its remedies.
• Amount evidence: 外滩大会线下圆桌｜敢把钱包交给AI吗？聊聊Agent交易爆发前夜的信任基建 closes by asking how much each participant would let an agent spend, linking higher limits to authorization, safety, and compensation confidence.

Counterevidence & Qualifications

• Familiarity can create complacency without actual safety, so repeated use is not sufficient evidence of calibrated trust.
• Institutional compensation can reduce user loss while leaving ranking bias, privacy harms, or merchant exclusion unresolved.
• The source offers directional frameworks rather than measured thresholds for when limits should increase.

What Changed

• Added a trust model that combines demonstrated ability, bounded autonomy, evidence, recourse, and progressive spending authority.

Related Concepts

• Agent Payment Infrastructure / 智能体支付基础设施 - institutional mechanisms that make transaction trust operational.
• Agent Spend Controls / 智能体消费控制 - graduated monetary boundaries used during trust building.
• Verifiable Intent / 可验证意图 - evidence of what the user actually authorized.
• Agent Permission Boundaries - limits that keep autonomy proportional to confidence.
• Human Agency Under AI - broader requirement that delegation preserve meaningful human control.