Following the recent PPA AI licensing report, this monthly series will further explore the impact of AI on the publishing and media sector.

Every publisher has spent the last decade building audience segments – and pricing and product decisions around them. That discipline now needs to expand. More than half of all traffic hitting the open web is non-human: bots now account for 57.4% of all traffic to HTML content, per Cloudflare’s own network data (early June 2026).

That number alone doesn’t tell you much – but it’s worth looking underneath: 

• AI training: 52% of crawler requests, up sharply from 22% in 2025. Scraped content disappears into the model: once used for training, it becomes part of the model’s internal weights and parametric knowledge – and unattributable.
  
  
• Mixed-use crawlers: 36%+ of activity. These blend search, live retrieval and training into a single bot (Google’s is the clearest example).
  
  
• Pure search: 8 – 9%, small and declining, despite remaining critical for publisher visibility. This is the closest thing left to the old referral-traffic exchange.

Each of these calls for a different commercial response. The pure search, and therefore mixed-use crawlers too, is where the commercial opportunity sits. When a chatbot or AI search product fetches a page to answer a live query, that’s a discrete, monetisable event.

Matching the monetisation model to the bot

Once traffic is segmented this way, “how do we monetise AI content” splits into three more answerable questions.

1. AI training – monetise via a bilateral licensing deal, or block entirely as a competitive moat. Built by OpenAI, Google, Meta, Amazon and Microsoft (direct deals) and Cloudflare (default-block infrastructure). Maturity: established, but concentrated among large publishers.
   
   

2. Search and mixed-use traffic – several distinct routes are emerging, each with a different mechanic:
   
   
   1. Pay-per-crawl
      a publisher sets a price per bot visit. TollBit and Cloudflare operate this at the crawl level; MonetizationOS installs directly on a publisher’s site as an access-and-billing layer, letting publishers allow, deny or charge bots by type, alongside their existing subscriber paywall logic.
      
      
   2. Pay-per-use
      Ceramic.ai crawls the open web, builds its own index, and sells access to AI companies as a search API – publishers opt in and are paid per query when their content is used in an answer, closer to a wire-service model than a licensing deal. Microsoft’s and Amazon’s content marketplaces work on a similar logic.
      
      
   3. Pay-per-appearance
      ProRata is building a licensing model priced on appearance in an AI answer, an approach that could offer publishers a more precise, usage-reflective way to be paid than crawl. The main challenge is verification: knowing how much of an answer actually drew on a given publisher’s content depends on visibility into the AI companies’ own usage logs, which publishers don’t yet have. Worth watching how this is resolved as attribution-based models like this one scale.
      
      
   4. Ad ingestion
      Blankspace sells an ad layer, placing brand-sponsored content into or alongside AI-mediated results and sharing that ad revenue with the publisher – similar in spirit to programmatic display, built for an AI-native surface.
      
      
   5. Content marketing/brand exposure
      A more classical, still-unproven approach – treating AI visibility like an earned-media channel rather than a direct payment.

Any of this can also be folded into a bilateral licence – but whichever route you choose, insist on transparency about the basis for payment: crawl, use, or appearance are three different things to be charged for.

Deal-based vs infrastructure-based: how they sit together

Deal-based licensing is a publisher negotiating directly with an AI company for payment to use its content. Infrastructure-based monetisation is a publisher using a third-party tool to price and charge for bot access automatically, without a negotiated relationship on the other end. The two can and do coexist – but for both, monitoring bot traffic is a hygiene requirement.

Bottom line for publishers

• Segment before you block. Training, mixed-use and search each need a different policy, a single crawler-level decision is too blunt now.
  
  
• Training is a one-off, binary call. Licence it or block it before the crawl. There’s no monetisation model after the fact.
  
  
• Mixed-use and live-retrieval traffic is where to experiment now, it’s the fastest-growing, most monetisable bucket, via pay-per-crawl or pay-per-use tools.
  
  
• Set revenue expectations low. Infrastructure-based monetisation won’t replace lost ad or subscription income yet – treat it as defensive recovery, not growth, until publisher leverage catches up.
  
  

Articles worth reading:

The curveball worth flagging this month: Google’s UK deals
Google is quietly signing major UK publishers – including the Guardian and FT – to confidential two-year AI deals worth single-figure millions, with NDAs attached, according to Press Gazette. One adviser called it a prisoner’s dilemma: sign, or lose out to a competitor who will, and mass take-up could render the CMA’s opt-out ruling moot before it’s ever tested. Press Gazette article.

New this quarter: the traffic-cannibalisation evidence just got a lot stronger
Traffic cannibalisation is no longer just correlation, it’s causal. An early 2026 randomised experiment (Agarwal/Sen, 1,065 Chrome users) found AI Overviews cut outbound clicks by 39.8%, with zero gain in user satisfaction, finally closing down Google’s “higher-quality clicks” defence. SSRN paper.