Cloudflare's managed robots.txt leaves ChatGPT search allowed

Quoted from Trakis: Block training, keep answers: Cloudflare offers a managed robots.txt…
Quote: Trakis, "Block training, keep answers"

Summary

Nearly half the sites that block OpenAI's GPTBot in robots.txt inherited that rule from Cloudflare's managed file, which leaves ChatGPT's search crawler allowed.

Fetch the robots.txt your server actually returns and decide whether OAI-SearchBot belongs in it by name.

Of the 1,331 sites that block GPTBot by name in robots.txt, 609 get that rule from Cloudflare’s managed file rather than from anyone at the site. The split comes from a study posted to r/TechSEO by the operator of Trakis, a small SEO audit tool.

The study, published on 2026-11-04, read 13,938 robots.txt files from Common Crawl’s September copies and counted what each asks of 14 crawlers. Cloudflare’s managed file lists training crawlers and names no AI answer bot, so OAI-SearchBot, which fetches pages for ChatGPT’s search answers, stays allowed. Where someone at the site wrote the GPTBot rule themselves, close to a third shut OAI-SearchBot out too.

Where the GPTBot block came fromSitesAlso block OAI-SearchBotAlso block ChatGPT-User
Cloudflare’s managed robots.txt6090.3%1.0%
Written at the site72231.3%47.0%
All sites blocking GPTBot1,33117.1%25.9%

The 17.1% row averages two groups that behave nothing alike. In the thread, the author says they nearly posted that figure on its own and only found the two populations after reading files by hand; the full write-up carries the per-bot counts. Any industry blocking rate quoted without the origin of the rule has the same problem, so it is worth little as a benchmark for your own settings.

Blocking the training crawler while leaving the answer crawler alone is a coherent position. The model stops learning from the site, and the site still gets fetched when a ChatGPT user asks something it can answer. The likelier explanation for most of those 609 sites, though, is that nobody weighed that trade: a vendor default produced the outcome a careful operator might have chosen, and from the file alone the two are indistinguishable. Sites that did write their own rules drew the line somewhere else again, blocking ChatGPT-User, the fetcher that pulls a page on a user’s request, 47.0% of the time. Three bot roles, three separate decisions, and Cloudflare’s toggle answers one of them.

Robots.txt is also not the only gate. In the thread, the author notes a site can allow OAI-SearchBot in robots.txt and block it at the firewall, and the study dropped 1,360 sites that refused Common Crawl their robots.txt at all, 782 of them behind a bot challenge. That gap between the stated policy and the served response is the same one behind bot protection blocking AI crawlers that robots.txt lets through, and the thread’s own conclusion is that server logs, not the file, settle it. The archived copies also held up against live reads: on 192 sites, the live file matched for every bot 92% of the time, and the mismatches were all on training bots.

What to do

  • Fetch the robots.txt your domain serves and find out who wrote the AI rules. If the GPTBot line in the browser is not in the file in your repository, it is being inserted at the edge, and whoever set that toggle made your AI search decision for you.
  • Treat the names separately rather than as one AI policy. The write-up’s list of answer bots to check by name: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. Platform controls increasingly follow the same split, as Vercel’s firewall does between AI crawlers and AI assistants.
  • Confirm in server logs that the bots you allow are getting 200s, since a WAF rule or bot challenge will block them whatever robots.txt says.

The data and code are on Zenodo under CC BY and MIT, so the per-bot counts can be rerun against a more recent crawl.