“Bad bot” bill heads to U.S. Congress as push to stop crawlers intensifies

By Robert Whitehead

INMA

Sydney, Australia

Connect      

The Stealth Bot Prohibition Act is set to be introduced to the U.S. Senate to spearhead a campaign to clamp down on all anonymous bots on the Web. 

Supported by both major political parties, the bill would require all crawlers, bots, and agents to identify themselves to Web site owners for each visit. The legislation is the first national bill of its type and other countries will need to replicate it for it to be effective.  

It arrives as the push to stop crawlers accelerates: Four tech giants committed to respecting requests to block AI training; a leading expert calls for drastic technical intervention to stop unauthorised access before payments for content can take off; and publishers are switching to blocking by default.

Here are the key points for publishers: 

The bad bot bill: The bill, positioned as transparency standards, would ban automated bots that hide their identity or purpose from accessing Web sites in ways likely to harm or disrupt the business. It would also ban disguising bots as human users in connection with generative AI, with U.S. authorities able to issue fines for breaches. 

Image from the NMA campaign library.
Image from the NMA campaign library.

The problem: Apart from content theft, publishers foot the direct cost for the server capacity and bandwidth consumed by bots crawling their sites, even when those bots send no readers or revenue back. The indirect cost for media companies is greater — scraped content used for AI training drastically reduces demand for commercial licensing. 

The supporters: The U.S. News/Media Alliance is campaigning for the legislation alongside a wider group of publishers, technology companies, and rights organisations and has garnered global backing.

Media companies include: News Corp, Advance Local, Axel Springer U.S., Condé Nast, Hearst Magazines, McClatchy Media Company, Newsmax, Tampa Bay Times, The New York Times, USA Today Co, Vox Media, and Reddit. 

The rationale: “Rather than defending against bad bots that now outnumber humans on their Web sites … publishers should be able to focus on creating quality content that enriches our lives and holds the powerful to account,” the CEO of News/Media Alliance, Danielle Coffey, told INMA.

“Government must step in to provide common sense guardrails that stop malicious actors from undermining the market, creating national security concerns, and overwhelming local and community outlets.” 

Can’t stop there: New York’s state legislature passed a similar law in June, setting the direction for the national law. Similar legislation will be needed in every country to make it effective in stopping crawlers. Many crawling companies go to great lengths to hide their country of origin, often appropriating local consumer identities. A large pool of them originates in countries hostile to the U.S. and western media outlets. 

The campaign: The News/Media Alliance has released materials to increase political pressure to pass the bill and to help other national bodies in other countries pursue the equivalent laws.

Tech assurances. Meanwhile, four of the biggest AI developers — Google, Microsoft, OpenAI, and Anthropic — pledged co-ordinated support for a shared mechanism that lets Web sites opt out of generative AI model training in one universal request.

Under a new proposal from the Internet Engineering Task Force’s (IETF) “AI Preferences working group,” publishers can opt out by placing a single policy token in standard robots.txt files (rather than configuring dozens of distinct crawler directives). For publishers, this streamlines opt-out management without sacrificing traditional search indexing.

Missing in action: Absent from the joint commitment was Meta, which at the time was defending its new commercial terms to allow AI training from all Web sites of businesses that use Meta platforms. This has since been paused after a global revolt by publishers and pushback by regulators. Meta is expected to sign up to the new IETF policy.  

Know your limits: The publisher preference will apply only to how the content is used for AI training, leaving out whether it can be crawled. It also omits search and other inference-time uses and does not cover content extracted before the policy. The new line of code continues a current weakness for media: It expresses a preference and is not a physical block.

From black to white: Leading publishers are switching from operating blacklists of bots to whitelists — specifying which crawlers are permitted (e.g. for advertising, search, licensing deals), as per INMA colleague Jodie Hopperton’s relentless advice.

People Inc now blocks everything except for its whitelist of 38 approved crawlers. The rest, according to Chief Innovation Officer Jon Roberts, are parasites. “We lost no revenue, and we lost no traffic.” He describes the current set-up of robot.txt as being as useful as a chocolate teapot.  

Raise the friction: Publishers will need more protection. “I think we’re going to need an Operation Warp Speed on bot defence at this point,” the Miso Co-founder and CEO Lucky Gunasekara, told INMA’s Media Tech & AI Week in San Francisco recently. “This is not a product problem. At a machine learning level, we might need new instrumentation and new technology to defend against this.”

The door is wide open: He later provided INMA with a summary of a machine-access audit of 250 leading subscription news sites in the UK and EU. Calls routed across 10 scraping services returned substantial article content from 240 sites, or 96 percent.

For each service, the proportion of sites it measured that returned content in the test run was: Exa 88%, Nimble 88%, Parallel 66%, Tavily 77%, Keenable 88%, Firecrawl 88%, TinyFish 79%, Seltz 83%, Bright Data 56%, and Ceramic 9%. 

Turn up the heat: “I want to say this just emphatically,” Gunasekara told the INMA audience in San Francisco. “We do need to put up a fight and raise the friction and the temperature as much as possible.”

Banner photo: Adobe Stock by Freshidea.

About Robert Whitehead