ALERT: Subscriber Acquisition Master Class early registration deadline is Friday, click here for agenda

The contract has replaced the crawler, now who writes yours?

By Earl J. Wilkinson

INMA

Dallas, Texas, United States

Connect      

For 25 years the open Web ran on an expectation. You let the crawler in, and the search engine sent you readers. Nobody signed anything. A U.S. court has now told publishers the bargain they believed in never existed in law, and Meta drew the obvious lesson: If an expectation is not an agreement, write an agreement.

That is what I wrote about on Sunday. From October 30, any company that uses Meta’s products for business purposes — a Facebook page, an Instagram account, an ad — authorises Meta to use “Web scrapers, bots, robots, spiders, crawlers and user agents,” and to create test credentials, to crawl the domains “associated with your products or services” and use what it finds. There is no opt-out box. The remedy for anyone who disagrees is to stop using Meta for business. I know of no news brand of any size without a business account at Meta. That is the point of the clause: it doesn’t need to name journalism, because journalism is inside everyone.

Meta has since removed the word “train” and narrowed the purpose to “advertising and business products.” Its lawyers presumably know what that phrase means; the terms don’t say. The week’s reaction was fast. By Tuesday the U.K.’s Competition and Markets Authority had written to Meta and scheduled a meeting; two German publisher associations representing more than 9,000 media brands asked the European Commission to stop the terms, calling them “blatant theft” and citing the Digital Markets Act; and the U.K. News Media Association’s Theo Bamber asked the question of the week: “Does Meta think we can’t read?”

Here is the structural point. Meta has licensed journalism from roughly a dozen companies with the lawyers and leverage to make it pay. For everybody else, it has written a clause. The six largest AI companies struck 109 licensing and data-access deals between July 2023 and March 2026 at an average of about US$24 million, according to Pradeep Gairola’s INMA report; CNN has just joined Google’s programme on undisclosed terms. This is a content market being built with a ceiling, and Meta’s terms are a bid for the rest of the world’s journalism at a price of zero, bundled into the cost of a Facebook page.

The INMA Media Tech & AI Week attracted 159 delegates from 19 countries in San Francisco. Conference delegates here are doing a group exercise.
The INMA Media Tech & AI Week attracted 159 delegates from 19 countries in San Francisco. Conference delegates here are doing a group exercise.

I write this blog from the INMA Media Tech & AI Week in San Francisco, with the conference proceedings playing on the speaker ready room TV screen. We are about 25% of the away through the conference, so much of this blog post is raw. In total, the week has drawn together 159 media and tech executives from 19 countries, including 31 on a study tour of 11 companies and another 20 product and audience executives from our top corporate members at OpenAI on audience growth.

The first of five questions INMA posed before the week was “Who controls access to journalism?” Read against Meta’s clause, and almost everything we heard was an answer to it.

What the crawler sends back

Start with the traffic. The San Francisco and Silicon Valley study tour was told that bot traffic now exceeds human traffic across the Web; that search crawlers still send roughly one reader back for every 11 to 15 crawls; and that training scrapers, the largest volume by far, send none. Asked whether publishers are yet earning anything from charging bots, one host said simply: “Not yet.” The first thing the group agreed on in its closing debrief was humbling. Nobody has the answers yet, the technology companies included; two of our hosts arrived with as many questions for publishers as answers.

The advice from the companies that see the most traffic was consistent: Block by default, let verified bots back in one at a time, and use your crawl data as leverage. Scarcity is what gets publishers paid.

The tour’s deeper theme was that context beats capability: The models are commodities, while a publisher’s private data, archive and judgment are not, and the clearest newsroom gains so far are on the input side — research, monitoring, routine production — not machine-written stories.

INMA study tour participants visit Reddit, which aims to keep the Internet human.
INMA study tour participants visit Reddit, which aims to keep the Internet human.

The same hosts admitted the hardest case is the newest. I said from the conference stage on Thursday that agents don’t yet identify themselves as acting for a person. An American news executive put the dilemma precisely on the tour: What do you do with a paying subscriber who sends an agent to fetch the day’s top stories? Nobody on the tour — not Reddit, which blocks 22 million attempts on its front door a day, not the companies that route the Web’s traffic — had a way yet to give an agent limited, temporary permission. A conference speaker put the practical version to the conference: Every page was built for a human, down to the button colour, and the same page must now serve an agent that doesn’t care about buttons. Start with your highest-traffic pages and work backwards.

Stanford University’s Maty Bohecek shared how the growing share of the Web is now machine-written.
Stanford University’s Maty Bohecek shared how the growing share of the Web is now machine-written.

Stanford’s Maty Bohacek gave the conference the reason the question matters. I’ll leave his figures to the room, but the theme was unmistakable: A large and growing share of the Web is now machine-written, people cannot reliably tell, and the machine reader and the human reader want opposite things — one wants everything, structured and factual; the other wants less, with a voice and a story. Writing one page for both serves neither.

The insight: Meta’s clause and the tour’s traffic data describe the same market: machines read most of what you publish and pay for almost none of it. A one-page policy for readers, bots, and agents is now a board document, and the conference’s standards session with SPUR, RSL and IAB Tech Lab is where the industry’s own paper is being drafted (more next week).

Scarcity needs solidarity

The study tour’s money conversation was blunt. AI companies pay only when journalism is scarce, and for any single large publisher a side deal always looks better than holding out. Each one weakens everyone’s leverage. One strategist predicted the bilateral deals won’t last more than two or three years.

The alternative was in the room. A publisher described the collective agreement that covers every publisher in his country with Google and Microsoft: three years of work, modest money but a legal baseline — and it held because nobody signed separately. The platform money, by contrast, is unreliable. Reddit’s licensing income is small next to its advertising and its first deals come up for renewal within a year; “two years is several generations in AI,” they told us. More than one host described licensing as thin, unpredictable, or a race to the bottom.

Meta’s threat — leave if you don’t like it — is one INMA members have already lived through. Canadian publishers didn’t leave Meta; Meta left them in August 2023 and hasn’t returned. The damage was real: almost half of news engagement vanished, and a third of local outlets went inactive. But the same case file holds the other half. Le Devoir lifted direct traffic from 22% to 26% of its total and cut monthly churn from 2.3% to 1.4% with an onboarding journey built in anticipation of the ban. The Walrus doubled its traffic. Meanwhile, Facebook’s share of referrals has fallen from 14% to about 6%, and its performance programme pays an average of about US$75,000 a page. Those are the numbers to put beside the clause.

The insight: Meta switched off news in Australia in 2021 and backed down in three days; it lost its seat at IAB Sweden when Swedish publishers acted together. Meta responds to publishers acting in concert in a way it never responds to publishers acting alone. The clause is an opening offer. Answer it collectively.

The reader is the hedge

One observation from the study tour’s debrief stayed with me. Participants felt the reader was missing from much of the week’s conversation, and several worried that the technology companies already understand publishers’ audiences better than publishers do. The group’s own corrective was to know your readers better than the platforms do, and to remember that people who pay for news have always been a small share of the population — and that is enough. As one U.S. local publisher put it, nobody subscribes to learn what the mayor said at a press conference. They pay for the backstory that comes from knowing the mayor.

INMA’s Kerstin Hasse interviews Substack’s Hamish McKenzie at the Media Tech & AI Conference. He argued the media economy has been rewritten by creators.
INMA’s Kerstin Hasse interviews Substack’s Hamish McKenzie at the Media Tech & AI Conference. He argued the media economy has been rewritten by creators.

Every host that was thriving pointed to direct relationships, and so did the conference. Substack’s Hamish McKenzie, interviewed by Kerstin Hasse, argued that the media economy has been rewritten by creators and that the next phase is networks of niche publications under a looser definition of who belongs to a brand. His sharpest line: In the age of agentic AI, “the e-mail list is the best insurance for the direct relationships that you could possibly have.” 

Greg Piechota interviews Dan Oshinsky on the future of newsletters at news media companies, a Webinar that attracted 220+ participants.
Greg Piechota interviews Dan Oshinsky on the future of newsletters at news media companies, a Webinar that attracted 220+ participants.

Greg Piechota’s Webinar Wednesday with Dan Oshinsky supplied the numbers:

  • Americans spend almost as much time on e-mail as on social media.
  • The median news outlet reaches 12% to 13% of households in its market with newsletters, the top quartile 30%.
  • And the AI inbox is about to rank newsletters like a podcast feed, so “just getting into the inbox isn’t going to be good enough anymore.”

TikTok’s Mac Bennet and Olivia Weiss described a platform where more than half of users now come for news and the 25- to 44-year-old cohort watches, looks for the article and sometimes subscribes. The operative tools are article links on every video and a new call-to-action sticker to send people home. And The Guardian, with 1.4 million recurring digital supporters, launched shared accounts this week — the direct relationship extended to the household.

The insight: The one asset no clause can scrape is a person who chose you. Reach without the second step — the habit on ground you own — is what the news business did on Facebook for 15 years.

Consent, the other way round

Wednesday’s workshop with OpenAI produced the mirror image of Meta’s clause. The group sketched two models for agents as an audience segment. In the first, a branded publisher relationship, discovery starts with a reader’s need — say, buying a home — an agent surfaces the specialist publisher, useful sample content earns the choice, and with the reader’s approval the agent arranges a trial or connects an existing subscription. In the second, an independent agent blends licensed intelligence from several publishers; attribution is limited, traffic minimal, and the value is licensing revenue. The warning was to keep the two apart: Growing an identifiable audience and being paid for the use of your knowledge are different outcomes.

Twenty INMA members participated in a closed-door OpenAI Audience Growth Workshop in San Francisco on Wednesday.
Twenty INMA members participated in a closed-door OpenAI Audience Growth Workshop in San Francisco on Wednesday.

Notice the word that runs through the first model: approval. Meta’s consent is manufactured by continued use; the workshop’s is given by a reader, one decision at a time. The same instinct showed up in London this week, where Reuters described a Model Context Protocol server that lets clients’ AI systems search its archive and live content on Reuters’ terms. That is a publisher writing the contract.

The insight: Consent, credit, compensation — INMA’s Basic Principles put it in three words a year ago, and Meta’s clause fails all three. The workshop’s model passes them. Distinctive expertise, agent-readable structure and permissioned data are what make a publisher the party that writes.

What is worth a contract

A contract is only worth writing if you own something worth licensing. INMA’s new report released Thursday, Jodie Hopperton’s 9 Newsrooms Put Gemini Notebook to Work, published Thursday, is the week’s best evidence of what that is.

INMA released “9 Newsrooms Put Gemini Notebook to Work” on Thursday, coinciding with the Media Tech & AI Conference in San Francisco.
INMA released “9 Newsrooms Put Gemini Notebook to Work” on Thursday, coinciding with the Media Tech & AI Conference in San Francisco.

Nine publishers spent the summer building products on their own journalism, and the conclusion deserves a reading at every board meeting: “The news organisation advantage is not the ability to generate more content. It is the ability to assemble distinctive, verified knowledge, and design a useful experience around it.” Mediahuis NL turned fragmented government pages into a guide for residents with earthquake damage and drew 176 users and 451 queries on day one. Daily Maverick built three notebooks on one source base — for readers, journalists and producers — because one set of instructions could not serve all three. Funke’s red-teaming exposed gaps in its election reporting and sent editors back to commission articles. The tour’s debrief reached the same conclusion from the other direction: AI can produce the routine chronicle of the news, but not original reporting, relationships, storytelling or local trust.

Sonali Verma interviewed Taneth Evans of the Wall Street in the recent Newsroom and AI Master Class.
Sonali Verma interviewed Taneth Evans of the Wall Street in the recent Newsroom and AI Master Class.

Greg Piechota made the commercial case this week: a news subscription has become a credence good, paid for on trust in a process the reader cannot see, so publishers must show the process — The New York Times with The Backstory, the Financial Times with its named experts, the Irish Independent with 15 of 25 home-page slots for signature journalism. Sonali Verma’s Newsroom and AI Master Class said the same from inside the newsroom: Article volume is dropping as publishers concentrate on what audiences will pay for. And Dietmar Schantin made the organisational point: All 50 of the largest news brands now bundle, but “bundling is a pricing decision, rebundling is an organisational one” — a community with its own audience, its own price, and an AI assistant built on a decade of its own reporting.

The insight: What Meta is trying to take for nothing is exactly what the report, the newsletters data, and the newsroom master class say is worth paying for: verified knowledge, a visible process, and a reader who knows the difference.

Before October 30

The study tour’s closing checklist and my Sunday checklist turn out to be the same list:

  • Audit the footprint: Every page, account and ad account, and every domain they link to, because the clause reaches whatever is “associated with” them, archive and paywall included.
  • Block by default: Disallow Meta-ExternalAgent, the crawler Meta itself describes as training its models, leave the link-preview agent alone, and measure the bot traffic with the tools INMA has pointed members to since 2025. Check your gates, because “test credentials” puts your registration wall in scope.
  • Price the relationship: put the referral number, the monetisation payment and your dependence on Meta’s ad tools on one page, beside the value of your archive; most boards have never seen those figures together. Negotiate against the default, because the new terms are now Meta’s opening offer in every country where it is talking to publishers. And organise locally, because local pressure works and global pressure would work more.
INMA study tour participants react to the four-legged robot called Spot from Boston Dynamics at the Hearst stop.
INMA study tour participants react to the four-legged robot called Spot from Boston Dynamics at the Hearst stop.

Then the study tour’s own additions: write the one-page policy for readers, bots and agents; speed up AI in routine work and measure cost per output; and look after the people, so the newsroom stays confident its work still matters. A Dutch editor closed the debrief on the note I would choose: AI journalism isn’t about producing more articles; it is about doing better journalism.

Strip away the company name, and this is what the week of the INMA conversation told us. Consent for the use of journalism is moving from the crawler to the contract. The platforms are writing theirs. The industry has started writing its own, in standards, in collective deals, in agents that ask the reader first. So: What is your Web site worth to Meta? What is Meta worth to you? And who, on October 31, will be holding the pen?

About Earl J. Wilkinson