Bot protection blocks AI crawlers that robots.txt lets through
Summary
Bot-protection rules turned away a test crawler at a third of the large German shops whose robots.txt files allowed AI search crawlers, so an open robots.txt proves nothing about access.
Fetch a live page with each AI user agent you care about, record the status code, and take any rejection to whoever owns the WAF rules.
In a thread on r/TechSEO, a practitioner posting as DrKJING reports that 31 of 100 large German shops turned away their test crawler at the bot-protection layer. In robots.txt, only one shop out of 90 checked blocked OpenAI’s OAI-SearchBot.
The thread’s larger dataset is HTTP Archive, the public project that crawls millions of home pages each month and has stored per-user-agent robots.txt counts since February. DrKJING compared the 457,225 German home pages in the October run against the full 12.0 million. German pages name GPTBot at 4.6%, against 8.3% worldwide. Germany sits below the world average for each of the 13 AI user agents in the query. Ranked by traffic, the picture reverses: nearly half of the German sites in the top 10,000 of Chrome’s CrUX popularity ranking name GPTBot. Outside Germany that figure is 18.1%. The shops in the crawl test are large ones, and the large sites are the part of the German web that has made a choice about AI crawlers.
Where German sites do name an agent, they block the training crawlers and admit the retrieval ones. Of the sites naming each agent, the share whose rule is a block:
| User agent | Blocked where named |
|---|---|
| Bytespider (ByteDance) | 70% |
| meta-externalagent (Meta) | 64% |
| CCBot (Common Crawl) | 55% |
| GPTBot | 54% |
| PerplexityBot | 30% |
| OAI-SearchBot | 29% |
DrKJING counts a block as a Disallow with no matching Allow, and reports the same ordering worldwide. The four agents at the top of that table collect training data. PerplexityBot and OAI-SearchBot fetch pages to answer live questions, so blocking them costs visibility in AI answers rather than a place in a training set.
Among the 100 shops, then, robots.txt reads as open to OAI-SearchBot, and a third of them refused the request anyway. The thread does not say which products or rules did the refusing. The likelier explanation is that bot-management rules score requests on signals like IP range and TLS fingerprint, so they catch crawlers that no robots.txt line mentions.
A robots.txt file may not be a decision at all. Cloudflare’s toggle disallows eight crawlers, mostly training ones. Squarespace ships a file that lists 26 AI agents and blocks none. Among German Squarespace shops that shows up as 99.8% naming an agent and 0.2% blocking one. Cloudflare has also written its own AI rules into customer robots.txt files by default. Knowing which template produced the file comes before reading it as policy.
In one large shop DrKJING analysed, 83% of ChatGPT’s live fetches went to product pages, which matches server logs where ChatGPT’s visits concentrate on product pages. For a shop, an accidental block at the edge lands on the pages ChatGPT asks for most.
What to do
- Fetch a live page with each AI user agent you care about and record the status code. Take the exact string from the vendor’s documentation, since bot protection matches on it.
curl -s -o /dev/null -w '%{http_code}\n' \
-A '<exact user agent string from the vendor docs>' \
https://example.de/produkt/12345- Test the retrieval crawlers and the on-demand fetchers separately: OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User. A 403 from the edge shows up in none of the robots.txt testers.
- Take the list of rejected agents to whoever owns the WAF rules. An allowlist entry, not a robots.txt edit, is what clears an accidental block.
- If you query HTTP Archive yourself, compare the same home pages month to month. Worldwide pages naming GPTBot fell from 1.57 million in September to 1.0 million in October, mostly disposable domains sharing one template leaving the crawl.