For years, publishers managed bots with a block list: a running record of known bad actors, updated whenever someone noticed a scraper hammering the site. At INMA’s Media Tech & AI Week in San Francisco, that approach was declared out of date.

The advice from tech companies, publishers, and INMA’s own research leads was unanimous. Block by default. Decide which bots you actually need. Let in only those, on your terms.

“I think the unanimity on the bot strategy” was the week’s big surprise, said Robert Whitehead, INMA’s Digital Platform Initiative lead, during the closing Debrief podcast. “We’re getting very clear advice: Be very specific about what you let in.”

The black market for news

The case starts with who is actually visiting publishers’ sites. Cloudflare told study tour participants that agentic and other non-human traffic is now more than 60% of what it sees. And over at Amazon, Megha Kande, senior manager of product management for its CloudFront product, said bots and agents make up more than half the traffic on AWS’s delivery network.

Most of that traffic takes content without paying for it. Whitehead described a market that has long been trading in publishers’ journalism: “Language models are buying our data, just not from us.”

The old tools don’t stop it. Cloudflare compared robots.txt to the rules posted at a community pool, which anyone can read and anyone can ignore.

Matt Lindsay of Mather Economics, speaking from the audience, emphasised the importance of price. When content is free, AI companies “just use as much as they can possibly get,” he said. Setting a price is the critical step to rationalising that consumption.

That makes scarcity the goal. Cloudflare said the publishers it works with are shifting from block lists to “allow lists,” and that AI companies start asking to license content once they can no longer take it. One publisher went into five legal conversations with companies that had been scraping it. Three ended in content deals. The other two companies stopped scraping.

Eyal Knoll Wel Roth, monetisation in media EMEA industry specialist at AWS, put it bluntly on stage. Publishers should block any bot they can’t identify and make an offer to the ones they can. Asked how to get from today’s black market to a transparent one, he said publishers need to block everything so the black market is starved.

Blocking isn’t the strategy — terms are

Blocking alone isn’t the goal, though. James Henderson, founder and CEO of MonetizationOS, pointed out that most publishers either block most of their traffic or allow most of it, and neither approach sets the terms. He told the story of Bargain Finder, the first shopping agent, launched in 1995. The CD stores that blocked it didn’t stop people shopping online. They simply disappeared from its results.

The real question, Henderson said, is: “Who sets the terms?” That means recognising every visitor, human or machine, and holding each to an agreed deal.

Know which bots you need

“There are many good bots,” Whitehead said. “That’s why you’ve got to be careful.” His first caveat: “Make certain you don’t stop the advertising.”

Ads came up at AWS, too. One publisher said their attitude to risk flips when bots start firing ad requests and analytics events. The publisher would rather let a bot read the content than block a single real reader, but would rather block a bot than let it trigger a bad ad request. Kande said AWS can route suspected bots to a version of the page with no ad tags or analytics.

Search, in its current form, still matters. Kande said search crawlers send roughly one reader back for every 11 to 15 pages they crawl. Training scrapers send back nothing.

Then there are the bots a publisher may want for strategic reasons. Cloudflare distinguished between crawlers that collect content to train models and agents that fetch a page because a person just asked ChatGPT a question. Many publishers want to show up in the second without feeding the first. Some will admit an AI company they have a licensing deal with and block its competitors.

The agents your subscribers send

The hardest calls involve agents acting for paying readers. 

Julie Young, founder and CEO of Laurel, showed the conference the personal news feed she built with an AI agent that reads her Financial Times subscription for her.

If the FT blocked it, she said, “I would definitely unsubscribe.” An audience member noted that the FT’s terms of service prohibit exactly that kind of access.

In The Debrief, Whitehead put the question directly to his colleagues: Will publishers let in agents reading what their owners have paid for? Jodie Hopperton, INMA’s Product & Tech Initiative lead and curator of the study tour and conference, said yes. Whitehead said none of the publishers attending the INMA event in San Francisco were yet saying yes.

“We need to really close up the black market,” Hopperton said, “but we also need to be careful that we don’t cause friction for our users.”

The tools and their limits

The infrastructure for this kind of control is maturing quickly. 

Cloudflare keeps a directory of more than 800 verified bots that publishers can allow, block, or rate-limit by purpose. AWS lets publishers serve different versions of a page depending on who is asking, or answer a commercial bot with a payment request.

Stu Rogers, co-founder and CEO of startup Spread, summed up the model: Verify human visitors, allow the licensed agents you’ve done deals with, and block everything else.

None of it is airtight. Lucky Gunasekara, co-founder and CEO of Miso AI said his team had used scraping services to get past the defences of 250 of Europe’s largest publishers in minutes, for about US$20. He called it a Napster moment for the industry.

Scarcity only works together

The deeper opportunity comes only from a collective industry action. 

Kande said the buyer side is still weak, and that buyers won’t be interested until a critical mass of content sits behind a wall. Marcos Stupenengo, digital media executive and strategist at AWS, said that if the major trusted news organisations all blocked at once, AI companies would have no alternative but to pay. As long as a trusted source is giving its journalism away for free, he said, it’s hard to see why anyone would pay for it.

The study tour’s closing debrief reached the same point: Every publisher that leaves the door open weakens the scarcity everyone else is counting on.

Whitehead boiled down his advice to seven words: “Have a white list. Be very specific.”

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .