Robots.txt only stops AI crawlers that choose to obey it

robots.txt line: Disallow: /

Summary

Robots.txt rules are requests, so they only work on crawlers that choose to follow them. Scrapers that borrow names like GPTBot or Googlebot ignore those rules and can pile requests onto parameter URLs nobody wants crawled.

Verify a bot before trusting its user agent: reverse DNS for Googlebot, OpenAI's published IP lists for GPTBot. Rate-limit or block anything unverified at the CDN or server, and never throttle Googlebot.

A thread in r/TechSEO asking whether practitioners block AI crawlers in robots.txt turned up a problem robots.txt cannot solve. The original poster described AI crawlers ignoring crawl-delay directives and hitting low-value, parameter-heavy URLs hundreds of times a day. The user agents in the poster’s logs included bots presenting as ChatGPT and Claude, plus several Googlebot-like agents that did not resolve properly through reverse DNS.

The replies split three ways. Some practitioners block every AI bot. Others allow specific ones such as GPTBot and block the rest, and a few are holding off for fear of losing visibility in AI search later. One commenter allows only known, well-documented bots and rate-limits everything else. That commenter’s approach holds up best, because it does not depend on any bot choosing to obey robots.txt.

Robots.txt binds only the crawlers that agree to it

RFC 9309, the standard behind robots.txt, describes its rules as ones crawlers “are requested to honor” and states that they “are not a form of access authorization.” A Disallow line works only when the crawler reading it chooses to comply.

OpenAI says its crawlers comply. OpenAI’s crawler documentation says disallowing GPTBot indicates a site’s content should not be used for training, and it publishes the IP addresses each of its bots crawls from. The Reddit thread points at a different group: scrapers that borrow those names, and crawlers with no published policy at all. A scraper can put “GPTBot” in its user-agent header, and a robots.txt group for GPTBot cannot tell the real crawler from the impostor. The better bet is to treat robots.txt as a policy statement for crawlers that honor it, and to enforce anything that matters at the server or CDN.

The cost lands on URLs nobody wants crawled

Unwanted bot traffic tends to pile up on parameter pages, archives and faceted navigation. Each of those requests can still trigger a database query, even when the page ends in a canonical or a redirect. The Wikimedia Foundation saw the same imbalance at scale, with AI bots making up 35% of its pageviews but 65% of its most expensive requests.

Crawl budget is the second worry, and it is often misread. Other bots do not draw from a shared Googlebot quota. The risk is to server capacity: if non-Google bots slow response times, Googlebot may reduce its own crawl rate.

Checking who is really asking

A user-agent string proves nothing on its own, so verification follows each bot’s documented method. For Googlebot, that method is forward-confirmed reverse DNS (FCrDNS). Look up the hostname for the requesting IP, confirm it ends in .googlebot.com or .google.com, then look up that hostname and confirm it points back to the same IP. The Googlebot-like agents in the thread failed at the reverse DNS step.

For OpenAI’s bots, the check is the IP list. GPTBot’s ranges are published at openai.com/gptbot.json. A request that claims to be GPTBot from an address outside that list is a spoofer, and it belongs in a server or CDN block rule.

Verified AI bots then need different treatment depending on their job:

  • Training crawlers such as GPTBot and ClaudeBot collect model training data. Blocking them keeps content out of training and has no known effect on current rankings or on real-time retrieval.
  • Search and retrieval bots such as OAI-SearchBot and PerplexityBot power AI answers. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
  • User-action fetchers such as ChatGPT-User load a page when a person asks for it. OpenAI says robots.txt rules may not apply to them because a user started the action.

What to do

  1. Start in the server logs, not in robots.txt. Find which AI user agents hit the site, how often, and which URLs.
  2. Verify before trusting a user-agent string, with FCrDNS for Googlebot and the published IP lists for OpenAI’s bots. Block the failures at the server or CDN.
  3. Rate-limit unverified and unwanted bots at the CDN or application layer. Cloudflare, Fastly and other CDNs can cap requests per user agent or IP range, and nginx or .htaccess rate limiting is the fallback on shared hosting.
  4. Do not rely on Crawl-delay. The directive is not part of RFC 9309. Bing and Yandex honor it, and Google ignores it.
  5. Never rate-limit Googlebot or Bingbot. Googlebot already slows itself based on response times and HTTP 500 errors, and Google removed the Search Console crawl rate limiter in January 2024.
  6. Name each AI bot by its documented token when blocking in robots.txt, rather than tightening User-agent: *. Any bot without its own group falls back to the wildcard rules, and that includes smaller legitimate crawlers and monitoring tools.

OpenAI documents each of its crawler settings as independent of the others, so a site can stay in ChatGPT search while opting out of training. The file below stops only the real GPTBot. Requests that use the GPTBot name from outside OpenAI’s published IP ranges still need the server or CDN block from step 2.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /