Nine sites block GPTBot but let OpenAI's search bot crawl

Someone posting as u/atp_studio_coll to r/TechSEO resolved 19 AI user agents against the robots.txt files of 122 well-known sites and published the result agent by agent. The blocks cluster on crawlers that collect pages for training and thin out for the bots that fetch a page in real time because someone in a chat asked for it.

The post applied the rule the crawlers themselves apply rather than reading the files by eye. The longest matching user-agent group wins, and the wildcard group applies only when no named group matches. An agent counted as blocked only when its effective group carried Disallow: / with no Allow carve-out, so a site disallowing a few paths under * came out as partial. On that standard 35 of the sites block at least one AI crawler by name, and 23 allow every agent everywhere, among them Cloudflare, Netlify, Supabase, freeCodeCamp and Anthropic’s own site. Nine sites had no robots.txt at all, including Stack Overflow, LinkedIn, npm, Next.js and Tailwind.

Blocking concentrates on a short list of names.

Agent	Sites blocking it
Bytespider	30
ClaudeBot	28
CCBot	27
Meta-ExternalAgent	25
Applebot-Extended	24
GPTBot	23
PerplexityBot	22
Google-Extended	18

Those are training crawlers and search index bots. The user-triggered fetchers draw fewer blocks: ChatGPT-User is blocked by 15 sites, Claude-User and Perplexity-User by 18 each. Each of the three requests a URL only when a person asks a question that needs it, so on this evidence the least restricted way an AI system reaches a page is one user at a time. A log study on a new domain reached the same conclusion from the server side, where the first AI visitor was a live fetcher rather than a crawler.

Some of that gap is deliberate. Nine sites block GPTBot, which gathers training data, while allowing OAI-SearchBot, which fetches pages for ChatGPT’s search answers: Figma, Medium, Reuters, TechCrunch, Khan Academy, Airbnb, Tripadvisor, eBay and Canva. Writing a file that way takes two named groups and a working knowledge of which OpenAI bot does what, which is not something a block list accumulates by accident.

The split between vendors is harder to explain. Six publishers block ClaudeBot while allowing GPTBot: Washington Post, The Guardian, The Verge, WSJ, Wired and Ars Technica. Exactly one site runs it the other way, Khan Academy. The Reddit post stops at the counts and gestures at licensing without claiming it, and licensing is the likelier explanation than anything technical, since nothing about ClaudeBot costs more to serve than GPTBot.

The post counted a site’s /llms.txt only when the body came back as plain text, because 15 sites served their HTML 404 page with a 200 status. A separate status check of llms.txt files found nearly half were error pages. A check that reads the status code alone will report a file that is not there.

One person ran this in a single night with a checker they built and did not link, and partial blocks are excluded from every count, so treat the distribution as a signal rather than a census. Robots.txt is a request that plenty of crawlers ignore anyway.

What to do

Resolve your own file one agent at a time instead of reading it. A named group replaces the wildcard group for that bot, and the mistake runs toward less blocking than intended. In this invented example:

User-agent: *
Disallow: /

User-agent: GPTBot
Disallow: /private/

The wildcard group blocks everything there, and GPTBot is free to crawl the whole site except /private/, because its own group replaces the wildcard instead of adding to it. That is the mistake worth checking for first. Confirm the file returns a 200 with a plain text body while you are in there, since a 404 page served as 200 is a file no crawler can use.

Then decide the training and retrieval question on purpose rather than by incident. The vendors use separate names for separate jobs, so a site that wants out of training sets and into AI answers has to list them separately: GPTBot and ClaudeBot collect training data, OAI-SearchBot and PerplexityBot feed AI search results, and ChatGPT-User, Claude-User and Perplexity-User fetch a page when a user asks for one. Blocking the first group does nothing to the third. After that, server logs are the only record of which bots honoured any of it.