News Publishers, AI Scraping, and the Future of the Wayback Machine

概览

This episode of Marketplace Tech examines why some news publications are blocking the Wayback Machine’s crawlers. The core concern is that AI companies could use the Internet Archive’s publicly available snapshots as a route to access copyrighted news content for model training.

Andrew Deck of Harvard’s Nieman Lab says publishers are acting mostly preemptively, without direct evidence that their content has already been scraped through the Wayback Machine. But he frames the fear as rooted in earlier experiences where large language models were trained on journalism without upfront permission or compensation.

The discussion follows a broader tension: the Wayback Machine has long served a public-interest role in preserving the web, but AI scraping has changed the perceived cost-benefit calculation for publishers. The episode closes by asking what happens to public memory, accountability reporting, and Internet history if key publications are no longer archived.

分段落总结

[00:01] Preserving Internet History Is Getting Harder

[事实] The episode opens by stating that preserving the history of the Internet is becoming more difficult. [事实] Marketplace Tech introduces the Wayback Machine as an Internet Archive project that sends crawlers to capture snapshots of web pages. [事实] Some news publications are now blocking those crawlers because they worry AI companies could access the archive and train models on their content.

[00:43] Why News Content Matters to AI Companies

[事实] Andrew Deck says news archives contain large amounts of informational, edited text. [事实] He says this type of content has been used to train many large language models and commercial products built on top of them. [推测] The episode presents publisher archives as economically valuable data, not just public information.

[01:09] Publishers Are Acting Preemptively

[事实] Deck says publishers he spoke with were blocking the Wayback Machine largely out of fear of proxy scraping. [事实] He says none of the publishers could point to a specific AI company or direct evidence that this had already happened through the Wayback Machine. [事实] He adds that publishers feel burned by earlier large language model development, when large portions of their content were used without upfront compensation or permission.

[01:59] The Old Tradeoff Around the Wayback Machine

[事实] Deck says the business side of news publishing was never especially enthusiastic about the Wayback Machine. [事实] The tool has long been used by readers to get around paywalls and access for-profit journalism for free. [事实] He says there was an unspoken pact that the small financial losses were worth the social good of maintaining a public record.

[02:39] AI Changes the Cost-Benefit Calculation

[事实] Deck says news organizations often share values with the Internet Archive, including transparency, public access to information, and historical documentation. [事实] He says publishers are recalibrating because the risk is no longer just a reader bypassing a paywall. [事实] Publishers now worry that generative AI companies could use copyrighted stories to train foundation models or feed commercial chatbots. [推测] The arrival of AI has made a previously tolerable public-access compromise feel commercially risky to publishers.

[03:39] The Wayback Machine at a Watershed Moment

[事实] Deck says the Internet Archive is at a watershed moment. [事实] He describes it as a victim of eroding Internet social contracts caused by the explosion of AI crawlers. [事实] He says earlier web-crawling norms operated more like an honor system, where major Internet companies would usually respect publisher restrictions. [事实] He argues that AI companies have broken many norms of the open web.

[04:36] A Fragile Record of the Internet

[事实] Deck says it is concerning when many important publications are not included in the Wayback Machine’s record. [事实] He says the situation highlights how fragile Internet history is when it depends on a small number of nonprofit archivists. [推测] The episode suggests that decentralized or underfunded archiving may be inadequate for preserving a reliable record of the modern web.

[05:17] Possible Alternatives and Their Risks

[事实] Deck notes that the Library of Congress has a smaller but substantial Internet archiving project. [事实] He says state-funded archiving could be an interesting model. [事实] He immediately raises the concern that governments might try to change the historical record. [事实] He cites the Wayback Machine’s usefulness to journalists tracking removed or stealth-edited federal agency pages during the Trump administration.

[06:09] What Publishers May Lose by Blocking Archives

[事实] Deck says the Wayback Machine will not stop having value or purpose. [事实] He agrees with Mark Graham’s view that when news publishers turn away from libraries, they can hurt themselves. [事实] He says publishers may lose evidence, documentation, and tools that journalists use to do their jobs. [推测] The episode presents an irony: blocking the Wayback Machine may protect publishers from AI-related risk while weakening journalists’ own reporting infrastructure.

[07:06] Closing Note on Journalistic Reliance

[事实] Stephanie Hughes identifies Mark Graham as the director of the Wayback Machine. [事实] Hughes says many journalists, including herself, have relied on the Wayback Machine in their reporting. [事实] The episode credits Jesus Alvarado as producer.

[07:29] Promo for How We Survive

[事实] The transcript ends with a promo for How We Survive, a podcast hosted by Amy Scott about climate solutions. [事实] The promo mentions geoengineering ideas such as stratospheric balloons and space-based sunshades.

播客点评/总结

[推测] This episode is valuable because it explains a technical conflict in practical institutional terms: AI scraping is not only a technology issue, but also a business, copyright, archival, and public-accountability issue.

The strongest part of the discussion is its balance. It notes that publishers lack direct evidence of proxy scraping through the Wayback Machine, while still explaining why their fear is credible after earlier AI training practices.

[推测] A limitation is that the episode does not include direct comments from the Internet Archive, AI companies, or specific publishers in the transcript. It therefore frames the issue mainly through Deck’s reporting and interpretation.

[推测] This episode is best suited for listeners interested in journalism, AI policy, copyright, digital preservation, and how the open web’s informal norms are changing under commercial AI pressure.