Existential Risk Probability Estimation

Definition

Existential risk probability estimation is the practice of assigning numerical likelihoods to catastrophic future outcomes so that uncertain risks and interventions can be compared.

Current Synthesis

The episode presents P(doom)-style estimates as a learned decision habit associated with Effective Altruism, not as direct measurements. Numbers can force assumptions into the open and support expected-impact comparisons, but apparent precision can exceed the available evidence when outcomes are unprecedented, mechanisms are disputed, and calibration data are scarce.

Key Claims

• Numerical estimates let decision-makers combine an outcome’s magnitude with its believed likelihood.
• Effective-altruist training can normalize assigning probabilities even to extreme or highly abstract events.
• Back-of-the-envelope calculation may summarize substantial thought without becoming empirical validation.
• Precise percentages can alarm outsiders because they sound more measured than their evidence base may warrant.
• The usefulness of an estimate for comparing actions is distinct from its accuracy as a forecast.

Evidence

• Community practice: Inside the mindset of AI safety workers says numerical probability assignment is common in the AI safety community Kaplan studied.
• Training context: Inside the mindset of AI safety workers reports that Kaplan practiced estimating extreme events during an eight-week effective-altruism course.
• Epistemic limit: Inside the mindset of AI safety workers records Kaplan declining to assess the accuracy of a particular extinction estimate.

Counterevidence & Qualifications

The source provides no model, inputs, track record, confidence interval, or independent calibration for a specific P(doom) number. It explains why people quantify uncertainty but cannot show that a quoted percentage is more reliable than qualitative concern or scenario analysis.

What Changed

• Created the concept to separate numerical decision practice from empirical forecast validation.

Related Concepts

• Effective Altruism - movement context in which the episode locates the practice.
• AI Doomerism - debate in which extinction probabilities become politically and emotionally salient.
• AI Safety Worker Culture - community setting that normalizes the estimates.
• Risk Perception - broader process through which people interpret uncertain danger.
• Fat-Tail Risk - adjacent problem of low-frequency outcomes with very large consequences.