Don't hate the replicator, hate the game

Summary

This Planet Money episode follows Abel Brodeur and his attempt to make social-science research more reproducible through the Institute For Replication and the Replication Games. It turns the Replication Crisis into an incentive-design story: researchers face pressure to publish novel significant results, and that pressure can reward P-Hacking, weak documentation, and fragile findings.

The episode’s strongest contribution is its practical enforcement model. Replication packages, Preregistration, and transparency rules help, but Brodeur argues that research behavior changes more when authors believe independent teams may actually inspect code, data, and robustness.

Key Claims

• Abel Brodeur’s smoking-ban project exposed how easy it is to keep changing an analysis until it clears a Statistical Significance Threshold.
• The source treats P-Hacking as an incentive problem, not only as individual misconduct: journals prefer novel significant results, while null findings are harder to publish.
• Brodeur and colleagues found a cluster of published results just on the statistically significant side of the 5 percent threshold, consistent with publication filtering, analytical tweaking, or both.
• Journal requirements for replication packages and preregistration improve transparency, but the episode argues that they are not enough if no one checks the underlying work.
• The Institute For Replication gave Brodeur an institutional identity that made authors more likely to share data and code.
• Replication Games turn research checking into a hackathon-like event where teams reproduce code, inspect data, and then run Robustness Checks.
• The Montreal game had 71 replicators and 14 main teams; most papers had no major issue, while two teams found more serious problems.
• The episode distinguishes simple reproduction from robustness analysis: rerunning author code is more objective, while judging alternative specifications requires domain and methodological judgment.
• One government-trust paper had documentation and missing-variable problems, showing how supposed raw data and actual analysis files can diverge.
• A cartel paper’s result became insignificant when one cartel was removed, but the original authors argued that the removed cartel was central to the hypothesis.
• Brodeur gives authors a chance to respond before serious findings become public, which makes replication less accusatory and helps protect junior replicators.
• The episode frames post-publication errors as a system failure because peer review and journal processes should catch documentation, code, and robustness problems before citation and dissemination.
• Research Integrity Incentives are the central fix: the expected chance of being checked may matter more than harsh punishment after a problem is found.

Key Quotes

“IRS for academia” - the show’s analogy for credible research inspection.

“Star Wars: The Empirics Strike Back” - Brodeur’s published paper title.

Connections

• NPR and Planet Money - network and show context.
• Abel Brodeur, University Of Ottawa, Institute For Replication, and Replication Games - central person, institution, and program.
• Replication Crisis, P-Hacking, Publication Bias, Statistical Significance Threshold, and Research Integrity Incentives - main methodological and incentive branch.
• Replication Package, Preregistration, Robustness Checks, and Crowdsourced Academic Auditing - practical reform and audit mechanisms.
• Scientific Self-Correction, Scientific Skepticism, Observation Before Inference, and Asymmetric Information - broader evidence and information-asymmetry frame.
• Economist Trust Crisis, Expert Trust Repair, and Economic Way Of Thinking - professional credibility and economics-reasoning connections.

Contradictions

• No direct contradiction with existing wiki pages found.
• The source complements Economist Trust Crisis by shifting the trust problem from public communication to the internal credibility of published economics and social-science research.
• The cartel-paper example remains an unresolved interpretation dispute rather than a settled flaw: the replicators treated removing one cartel as a robustness check, while the original authors said that cartel was the intended central case.