OpenAI pays Cloudflare for hints on which pages changed

OpenAI is paying Cloudflare for hints

Summary

OpenAI is paying Cloudflare for signals that tell its crawler which pages have changed, Cloudflare CEO Matthew Prince said on the third-quarter earnings call, the first revenue Cloudflare has disclosed from making AI crawling more efficient.

Open AI Crawl Control, see which AI crawlers are reaching the origin, and set an allow or block rule for each one.

OpenAI is paying Cloudflare for hints that tell its crawler when a web page has changed, Cloudflare co-founder and CEO Matthew Prince said on the company’s third-quarter earnings call, per Search Engine Journal’s report. The same call marked Cloudflare’s first revenue from what it calls Act 4, the products meant to build a new business model for the internet, and Prince gave no figure for it.

The arrangement is the research pilot Cloudflare made public on July 8, which uses signals from participating websites such as content freshness and page changes. Prince’s example was a restaurant updating its prices: Cloudflare tells OpenAI the page is different, so the crawler does not have to keep going back to re-fetch it. He did not say what OpenAI pays or when the payments started, and he expects other AI companies to buy the same thing. He also said “OpenAI’s crawler” without naming it, so whether the hints spare GPTBot, OAI-SearchBot or both is not on the record.

The payment is for crawl efficiency, not for the content itself. Charging crawlers directly is further behind: in a test of charging AI agents per page, the billing worked and no search crawler could pay. The first money Cloudflare has confirmed in this market went to the network that told a crawler which pages it could skip. Whether the sites whose freshness signals feed the pilot get a share was not addressed on the call.

Prince’s argument for charging someone is volume. An agent choosing where its user eats dinner might pull menus from 1,000 restaurant sites, he said, and the user goes to one. The rest absorb the cost of a request with no customer at the end of it. On Cloudflare’s August call he put nonhuman traffic at up to 1,000 times human traffic within five years, and on this call he said he expects that sooner. “It can’t be a free ride for all the agents that are out there,” he said.

Customers split into two camps by Prince’s account, some wanting as many agents as possible and others who “see them as a bane to their business and want to keep them away.” For the first group he called what Cloudflare does “sort of the new version of SEO,” and then named no product that does it. The agent-facing products that exist are per-crawler blocking and two payment betas opened on Sept. 30, the Monetization Gateway and Pay Per Use, covered in Cloudflare’s test of charging AI agents instead of blocking them. Prince expects Cloudflare to say more about Act 4 “in the quarters to come,” so agent visibility is a promise rather than something to configure this month.

He also expects micropayments, with cheaper access for agents that explain why they came. An agent that tells a travel site it is planning a Hawaii trip for a family of four might get in free, in his example, because the request opens up advertising. Prince is not sure how long it will take for that to become a major source of revenue. None of it is configurable today, which leaves allow-or-block as the one decision a site can make now.

What to do

AI Crawl Control is available on every Cloudflare plan, shows which AI crawlers are requesting pages from the origin, and sets allow or block rules per crawler. Pull that list before deciding anything, because the right answer differs by bot type:

  • Training crawlers such as GPTBot and ClaudeBot send no visitors back, so blocking them costs no traffic.
  • Search and retrieval bots such as OAI-SearchBot and PerplexityBot feed AI answers that can cite the site, and blocking them takes it out of those answers.
  • User-triggered fetchers such as ChatGPT-User and Claude-User represent a person asking for that specific page, which makes them the hardest to justify blocking.

Prince said customers who want agents kept away are not charged for the blocked traffic, so blocking is also the cheaper option for a site seeing no return from agents. Cloudflare’s docs list Pay Per Crawl as a private beta, so charging per crawl is not a lever most sites can pull yet.

Watch out for

A fall in OpenAI crawl volume at the origin is not a sign the site improved. Under this deal it may mean Cloudflare told the crawler the page had not changed. The disclosures to watch are whether other AI companies start paying for the same hints, as Prince predicted, and whether Cloudflare ever says how much of that money reaches the sites supplying the pages.