Last month we made the case for bot segmentation: training crawlers, live retrieval bots, and search crawlers each carry different commercial value, so treating them as all the same is a strategic mistake. Catch up here.

This month, the harder question: how do you actually enforce the strategy and make that segmentation work for the publisher?

A bot is, in effect, a computer programme. By default it doesn’t know whether the publisher has signed a licensing deal, or what their strategic intentions are. It needs this information in a format it can parse. So the whole strategy depends on the two layers working together:

• Legal protections – what you’re entitled to demand, and how you enforce it when someone ignores you
• Technical protections – what actually happens at the point when a bot accesses your servers to request your content

Layer 1: Legal – declare granting or blocking access, then be ready to act

Robots.txt, Really Simple Licensing (RSL) and the new Search-Only Contract (SOC, by the Movement for an Open Web) are all ways of saying the same thing in a language bots understand: here is what bots can and cannot access.

FT Strategies’ analysis of 70 major publishers’ robots.txt files shows clear patterns: 39 block GPTBot (training bot) while only 16 block OAISearchBot (live answers bot). Publishers are welcoming the crawlers that potentially send traffic back to them (e.g. live answers and search) and closing the door on the ones that just absorb the archive (training).

Robots.txt has 30 years of crawler convention behind it, however imperfectly followed (per the crawl-ratio data: Anthropic 20,583:1, OpenAI 1,255:1). RSL is brand new and expresses much richer terms. Robots.txt is a binary allow/block switch with no way to express payment, while RSL layers licensing terms, including per-use pricing and machine-payable compensation, on top of it.

However, no major AI company has yet agreed to honour those terms or pay under them. OpenAI, Google, Meta, and Anthropic have all declined to comment on compliance. So today, RSL is a declaration and a future negotiating position.

The Search-Only Contract is where it gets interesting. The Movement for an Open Web launched with 31 publishers and was backed by the PPA. SOC actually has enforcement built in: a £500-per-article claim against unauthorised scraping, using contract law rather than copyright law as the route. Proof is obtained via targeted questioning of the chatbot itself, and unpaid invoices head to the small claims court. This is the first mechanism in this space designed to enforce payment, not just make a declaration.

The risk of making these declarations more confidently has been reduced by the CMA’s Publisher Conduct Requirement on Google. Live since June 2026, it lets publishers opt out of AI Overviews and AI Mode without losing traditional search ranking – previously the coercive trade-off that made opting out unthinkable. The onus, though, is still on the publisher: declaring which bots get access and which do not remains the publisher’s job.

Layer 2: Technical – where declaration can be enforced

It is possible to hard block all bot traffic at the CDN level, but this blanket blocking forgoes opportunity as much as it prevents harm. Block everything and you may also block the retrieval bots that could, under the right deal, drive discovery back to you. This is why alignment is critical: robots.txt, the CDN rules, and your rights declarations need to say the same thing, or a bot will simply do whatever the weakest signal allows.

Cloudflare now defaults to blocking mixed-use AI crawlers on ad-supported pages. Like Cloudflare, Akamai also reports AI bot traffic increases – up by 300% in 2025 – and is moving publishers from blunt blocking towards selective blocking.

Every barrier we build is also a signal to readers: this is where the original content lives, something they cannot get anywhere else. So the core moat remains: Making the path to original journalism as frictionless as the one AI chatbots offer is going to be the competitive advantage.

Worth watching: cross-industry watermarking under the EU AI Act (effective August 2026) is starting to distinguish AI-generated content from the real thing – a provenance signal publishers can use on their own terms, not just defend against.

From other news

Speaking of provenance, NYT’s Hard Fork podcast recently sat down with Max Spero, CEO of Pangram, whose AI-text detector has become something of a breakout hit by exploiting the subtle, aggregate patterns in how models sample tokens. It’s not protection in the licensing sense, but it’s the same instinct: knowing what’s machine-made and what isn’t matter to readers, not just to publishers chasing compensation for their work.