As AI shifts from crawling to citing, the page stops being the product
Product and Tech Blog | 11 August 2026
Almost every conversation I have with a publisher right now opens with the same number. Someone pulls up a traffic chart, points at the month the line turned, and asks what can be done about it.
It is an entirely reasonable place to start. It is also, I have come to think, the wrong number to be planning around.
The decline is real and I am not going to argue otherwise. But it has been the loud half of the story for two years now, and while the industry has been watching it, the quieter half has moved considerably further.

The decline is real, but it is not uniform
The number most people quote comes from Chartbeat, working with the Reuters Institute: Google search referrals to publishers fell roughly 33% globally in the year to November 2025.
The one that should actually worry this audience came in March, when Chartbeat broke the same data out by publisher size. Over two years, small publishers lost about 60% of their Google referral traffic. Medium publishers lost about 47%. Large publishers lost about 22%.
That spread is the whole problem in one statistic. The organisations with the least capacity to rebuild took nearly three times the damage of the ones with the most. If you run a mid-market group, the industry average was never your number.
There is a less-discussed possibility worth holding onto, which is that the line may have flattened. Datos found the U.S. zero-click rate fell from 24.5% in December to 22.4% in March. I would not write a plan off a single quarter. But a 2027 budget written on the assumption that traffic falls indefinitely and one written on the assumption that it has reset to a lower, stable level are different documents, and only one of them frees up capital to build anything.
The quieter half changed in July
On July 1, 2025, Cloudflare told AI companies it would pay each time a crawler fetched a page. This was a serious attempt at a real problem, and it moved the conversation further in a year than the industry had managed in three.
On July 1, 2026, Cloudflare declared that model insufficient and replaced it. The successor framework compensates publishers when their content appears inside an answer rather than when it is retrieved. From September 15, its default settings will also block crawlers that serve both search indexing and AI retrieval from any ad-supported page, automatically, for every free-tier customer and every new site.
I care less about the mechanism than about the reasoning behind it. Search bots — the only category of crawler with even a theoretical path back to a human click — now account for roughly one-tenth of traffic to HTML content across Cloudflare’s network. Training-related crawlers account for half. Crawl-to-referral ratios for the leading AI bots run from about 118-to-one out to nearly 50,000-to-one.
Paying per fetch, in other words, was paying for the wrong event. Publishers spent 2025 arguing about who was allowed to take the content. The 2027 argument is about who gets credited when it is used. Compensation is moving from access to attribution, which is the same direction publishers were sketching out in Berlin in May, and those two things reward completely different work.
Licensing is the half that will not scale
The direct deals are real, and the people negotiating them are doing serious work under difficult conditions. OpenAI has built the broadest portfolio in the category, with more than 20 publisher partnerships. News Corp signed with Meta in March on terms reported at up to US$50 million a year.
The problem is not the deals. It is that this market pays negotiating leverage, and most of us do not have any. Litigation is not the alternative either. Most of the suits filed this year remain in discovery, and a plan dependent on a favourable ruling in a case you may never file is not a plan.
The development I would pay attention to instead got far less coverage. Wikimedia’s January arrangement with Amazon, Meta, Microsoft, Perplexity, and Mistral — and Reddit’s structure before it — are not archive sales. They are subscriptions to a continuously refreshed, well-structured feed. The buyers stopped purchasing a corpus and started renting a supply.
That distinction is the opening, because a well-structured feed is something a newsroom can build without anyone’s signature on anything.
What this actually asks of a newsroom
Two things, neither of which will make anyone want to sit through the budget meeting where you ask for them:
• The first is measurement. If compensation follows appearance in an answer, then citation position stops being a search metric and becomes a revenue input. Seer’s work this year found brands cited in AI Overviews earn roughly 120% more organic clicks per impression than uncited brands on the same queries.
I have yet to see it reported to a board alongside sessions and subscriptions. Whoever starts measuring it this year will be the only one in the room able to negotiate on it next year.
• The second is structure. Before content can be cited, priced, or retrieved by an agent, it has to be clean, tagged, and consistently described. This is infrastructure work with no launch moment attached — the kind of line item that gets cut first and misses second. It is also the precondition for everything above it.
INMA’s own agentic programming has arrived at the same conclusion from a different direction: Data has to be structured and ready before AI can deliver value against it. The organisations that will be able to price a feed in 2027 are the ones paying for the schema now, while it is still boring.
A structured archive is a floor, not a finish line
This is the part I tend to notice from the seat of someone who builds tools for newsrooms.
The teams recovering fastest are not the ones publishing more, which is the first thing most operators reach for. They are the ones that shortened the distance between a story being written and that story being structured and machine-legible.
In practice, this is less dramatic than it sounds: meta-data applied at the point of creation rather than retrofitted six months later, and one consistent schema applied to every asset regardless of who filed it. Where teams have done it well, pre-publish structuring work has fallen by more than half, and both traffic per story and revenue per story have moved in double digits.
The more telling detail is what happened to the time that freed up, because that is a chief operating officer decision and not a technology one. In the newsrooms that stalled, it was banked as headcount reduction. In the newsrooms that recovered, it went straight back into the data layer.
That single call, more than any tooling decision, separated the two groups. The tool is the input. A retrievable archive is the output, and the two are not the same thing.
What the rest of us should take from this
The traditional model is not collapsing so much as being repriced, and the repricing is happening on the supply side. Loyalty, subscriptions, and specialised content remain the right long-term bets, and this audience has been making them for years.
But loyalty is a demand-side strategy. What broke in 2025 and 2026 was infrastructure: how content is accessed, retrieved, attributed, and paid for by machines that will never see a paywall or a newsletter sign-up form.
So, when you sit down to write the 2027 plan, the question is not how much traffic came back. It is whether anything you published this quarter is in a form a retrieval system can use, credit, and pay for.
A page is what you publish. A record is what gets retrieved. Only one of those is still the product.
Banner image: Adobe stock image by Being Imaginative.








