Cloudflare feeds OpenAI's crawler pages it had not found

Cloudflare has been passing page-discovery signals to OpenAI’s search crawler since July 2026, and the pilot has put hundreds of millions of previously unknown pages into OpenAI’s search index. Cloudflare’s post on the early results, written with two OpenAI co-authors, says the signals help the crawler reach useful pages that few other sites link to.

The pilot ran on a participating cohort of about 1% of Cloudflare’s network. Cloudflare supplies signals about pages, and OpenAI evaluates and ranks them on its own. No new protocol came with the announcement. The post names no header and no user agent, and it does not say how sites came to be in the cohort or how a site would opt in or out of the signals themselves.

Cloudflare draws one line explicitly: a signal about a page does not grant permission to access it, and the pilot passes metadata about the web rather than page content for model training. That leaves a site’s existing crawl rules deciding everything. A URL can now reach OpenAI’s search crawler with no inbound link and no sitemap entry pointing at it. Submitting sitemaps was never much of a lever with these crawlers anyway, since AI crawlers only use the sitemaps they find on their own.

The post calls it the search crawler without naming a user agent, and OpenAI’s own documentation leaves one candidate. OAI-SearchBot surfaces sites in ChatGPT’s search features, GPTBot crawls for model training, and ChatGPT-User fetches a page when someone asks ChatGPT about it. OpenAI’s documentation gives the common setup as its own example: allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. Sites configured that way are the ones whose unlinked pages become discoverable first.

The signals cut fetches as well as add them. Cloudflare lists the waste it says it can now recognize before a crawler makes the request:

• bots asking over and over for files that are not there
• sites that redirect almost any address to the homepage
• addresses that lead to copies of the same content

Cloudflare’s post does not spell out what that means for a site owner, but a server answering unknown URLs with a redirect to its homepage is producing the pattern the filter discards. Cloudflare is also testing signals that flag single harmful pages, so a fake login form planted on one URL can be kept out of the index without the whole domain going with it.

Crawl volume can rise before it falls. Cloudflare says finding new pages initially increases crawling while those pages get evaluated and indexed, and that change signals are meant to stop repeat fetches of pages that have not changed.

What to do

• Check the per-crawler rules in AI Crawl Control, which shows which AI services access your content and sets allow or block rules for individual crawlers. The rule on OpenAI’s search crawler now governs pages you never submitted anywhere.
• Give anything that should stay out of an index a real reason to stay out: authentication, a 404 or 410, or a block rule. Being unlinked is not one.
• Fix catch-all homepage redirects and soft 404s on the URL patterns bots probe.
• Record current OAI-SearchBot hit counts from your logs now, so any increase over the coming weeks can be read against the pages it brings.

Watch out for

Obscurity stopped working as a control some time ago. A canary test already showed Grok hitting one unlinked page from 59 IPs, and this pilot turns unlinked discovery into infrastructure. At about 1% of the network, most sites will see nothing yet, and Cloudflare says the next phase will improve page selection, test faster change signals and measure what happens after content is discovered. What to settle before that expands is whether the crawl rules in place today are the ones you would choose for pages nobody has ever linked to.