Cloudflare tests charging AI agents instead of blocking them
Summary
Charging automated traffic is now a Cloudflare product rather than an experiment, but the pricing half is closed beta and limited to some US customers, so most sites cannot use it.
Usable today are the separate search, agent and training controls and the per-crawler reporting behind them. Work out which of your automated traffic has a person behind it before setting any of it to block.
Cloudflare has started charging for automated traffic rather than only blocking it. Its post on the agentic web announces a Monetization Gateway, in closed beta for eligible US customers, that puts a price on any request matching a rule, written in the same Rules language Cloudflare customers already use at the edge. A priced request gets back an HTTP 402 Payment Required carrying the open x402 payment protocol, and the agent pays the seller directly. Prices can be set per request, per query or per token, at fixed or capped rates.
Cloudflare’s own example is a sports statistics site funded by ads that charges a fraction of a cent each time an agent asks which player leads the league in assists. The same fact, fetched by software, now has a price on it.
Cloudflare moved off the training-crawler block it recommended last year because its two kinds of non-human traffic have stopped behaving alike. A training crawler collects pages to build a model. An agent returns every time someone asks a question that touches the content, so agent traffic grows with how many questions people ask rather than with how much a site publishes. Daily agent requests on Cloudflare’s network grew more than 1,700% over the past year. This year, for the first time, more than half of internet traffic was not human.
Training crawling grew too. Crawler requests stating a training purpose were 22% of the total in spring 2025. By June 2026 they were 52%.
The paying half is the part not to plan around yet. Closed beta for eligible US customers means most sites cannot price anything today, and an independent x402 test covered here earlier showed the harder limit: Googlebot and the other search crawlers cannot pay a 402 invoice. An unpaid 402 is a block. Pricing a URL you need indexed is a decision to lose the indexing, which makes the Gateway a tool for pages that exist to be queried rather than for pages that need to rank. Cloudflare’s other program, Pay Per Use, avoids that trap. Verified crawlers fetch pages as before and report when content is used, and Cloudflare bills the buyer and pays the publisher, who sees each offer and opts in or out.
What is available on every plan, including Free, is the split Cloudflare shipped in July: separate Search, Agent and Training controls in place of one “block AI bots” toggle. On September 15 it added Disallow AI Training, which keeps a site indexed for search while instructing the operator not to use its pages for training, and Apple, Google and Microsoft have committed to honor it. That covers the mixed-use crawler, where one bot fetches for both search and training and refusing one used to mean refusing both. The same September change made the plain Training block shut out Googlebot as well, so Block is no longer the safe default for training.
Two smaller pieces in the same post are about serving agents rather than billing them. Markdown for Agents lets an agent read a page without the styling meant for human eyes, and WebMCP lets a site expose its actions directly instead of leaving an agent to guess which button to press.
What to do
Measure before deciding anything. Cloudflare’s AI Crawl Control, Business Insights and BotBase reports show which crawlers arrive, what they take and which URLs they want most. Web Bot Auth, a scheme under which operators including OpenAI, Google and AWS sign their agents’ requests, separates a real agent from something wearing its user agent. A site that does not know its automated share cannot tell a lost customer from a free rider.
- Set the controls by purpose, not by bot name. Disallow training on content you do not want inside a model. Block agents only where an ad impression is the revenue, since an ad pays only when a person sees it. Leave search open.
- Use Disallow AI Training, not Training set to Block, to stay in search results without feeding training sets. Only Apple, Google and Microsoft have committed to honor the signal, so check the crawler reports after switching it on.
- Leave pricing alone for now. The Gateway is closed beta for eligible US customers, and the only URLs worth pricing are ones that do not need to rank.