Don't hate the replicator, hate the game

Summary

This Planet Money episode follows Abel Brodeur and his attempt to make social-science research more reproducible through the Institute For Replication and the Replication Games. It turns the Replication Crisis into an incentive-design story: researchers face pressure to publish novel significant results, and that pressure can reward P-Hacking, weak documentation, and fragile findings.

The episode’s strongest contribution is its practical enforcement model. [[ReplicationPackage|Replication packages]], Preregistration, and transparency rules help, but Brodeur argues that research behavior changes more when authors believe independent teams may actually inspect code, data, and robustness.

Key Claims

• Abel Brodeur’s smoking-ban project exposed how easy it is to keep changing an analysis until it clears a Statistical Significance Threshold.
• The source treats P-Hacking as an incentive problem, not only as individual misconduct: journals prefer novel significant results, while null findings are harder to publish.
• Brodeur and colleagues found a cluster of published results just on the statistically significant side of the 5 percent threshold, consistent with publication filtering, analytical tweaking, or both.
• Journal requirements for [[ReplicationPackage|replication packages]] and [[Preregistration|preregistration]] improve transparency, but the episode argues that they are not enough if no one checks the underlying work.
• The Institute For Replication gave Brodeur an institutional identity that made authors more likely to share data and code.
• Replication Games turn research checking into a hackathon-like event where teams reproduce code, inspect data, and then run Robustness Checks.
• The Montreal game had 71 replicators and 14 main teams; most papers had no major issue, while two teams found more serious problems.
• The episode distinguishes simple reproduction from robustness analysis: rerunning author code is more objective, while judging alternative specifications requires domain and methodological judgment.
• One government-trust paper had documentation and missing-variable problems, showing how supposed raw data and actual analysis files can diverge.
• A cartel paper’s result became insignificant when one cartel was removed, but the original authors argued that the removed cartel was central to the hypothesis.
• Brodeur gives authors a chance to respond before serious findings become public, which makes replication less accusatory and helps protect junior replicators.
• The episode frames post-publication errors as a system failure because peer review and journal processes should catch documentation, code, and robustness problems before citation and dissemination.
• Research Integrity Incentives are the central fix: the expected chance of being checked may matter more than harsh punishment after a problem is found.

Key Quotes

“IRS for academia” - the show’s analogy for credible research inspection.

“Star Wars: The Empirics Strike Back” - Brodeur’s published paper title.

Connections

• NPR and Planet Money - network and show context.
• Abel Brodeur, University Of Ottawa, Institute For Replication, and Replication Games - central person, institution, and program.
• Replication Crisis, P-Hacking, Publication Bias, Statistical Significance Threshold, and Research Integrity Incentives - main methodological and incentive branch.
• Replication Package, Preregistration, Robustness Checks, and Crowdsourced Academic Auditing - practical reform and audit mechanisms.
• Scientific Self-Correction, Scientific Skepticism, Observation Before Inference, and Asymmetric Information - broader evidence and information-asymmetry frame.
• Economist Trust Crisis, Expert Trust Repair, and Economic Way Of Thinking - professional credibility and economics-reasoning connections.

Contradictions

• No direct contradiction with existing wiki pages found.
• The source complements Economist Trust Crisis by shifting the trust problem from public communication to the internal credibility of published economics and social-science research.
• The cartel-paper example remains an unresolved interpretation dispute rather than a settled flaw: the replicators treated removing one cartel as a robustness check, while the original authors said that cartel was the intended central case.