P-Hacking

P-hacking is the practice described in Don’t hate the replicator, hate the game where researchers repeatedly adjust data choices, model specifications, sample definitions, or other analytical decisions until a result clears a Statistical Significance Threshold. The episode grounds the problem in Abel Brodeur’s smoking-ban project, where repeated reshaping could have produced a publishable effect even though his straightforward analysis found none.

The concept is not presented as only fraud. The source emphasizes that Publication Bias and career pressure can make small defensible-looking choices accumulate into misleading certainty, especially when journals reward novel significant findings more than null or messy results.

Key Claims

• P-hacking exploits analytical degrees of freedom around data, variables, samples, controls, and models.
• It becomes more tempting when a null result is hard to publish.
• A result can look statistically official while depending on a sequence of researcher choices that readers cannot easily see.
• [[ReplicationPackage|Replication packages]] and Robustness Checks expose p-hacking risk by making the underlying workflow inspectable.
• Preregistration can reduce p-hacking by requiring hypotheses and analysis plans before the researcher sees all results.

Connections

• Abel Brodeur - source case and researcher who turned the experience into a broader research agenda.
• Replication Crisis - wider problem that p-hacking contributes to.
• Publication Bias, Statistical Significance Threshold, and Research Integrity Incentives - incentive environment around the practice.
• Scientific Skepticism and Rational Humility - broader reasoning guardrails.