Grok hit one unlinked page from 59 IPs posing as Chrome or Safari
Summary
User-agent blocklists only stop the fetchers that identify themselves honestly. In one Reddit test, two assistants arrived under names that no published IP list backed up, and one spread each request across dozens of addresses at hosting companies and small ISPs.
The cheap check is one unlinked URL per assistant, logged at the edge. It costs an afternoon and tells you who actually fetches your pages instead of answering from memory.
A Reddit user gave each AI assistant its own secret URL, put a Cloudflare Worker in front of it to log every request, then asked each assistant in a fresh chat to open its link and read back a code printed on the page. Nothing anywhere linked to those URLs, so whatever hit one had to be the assistant that was handed it, and a correct code read back proved a live fetch rather than a guess. The r/TechSEO thread, posted by u/hunxai69, who also published the method and the raw logs on GitHub as whofetched, reports that the assistants split into the ones a server can recognise and the ones it cannot.
What landed in the logs differed by assistant:
| Assistant | What arrived | Read robots.txt |
|---|---|---|
| Claude | a single IP, self-identified | yes, before the page |
| ChatGPT | the page first, OAI-SearchBot afterwards | yes, about a minute later |
| Gemini | user agent “Google”, IPs off Google’s published lists | no |
| Grok | 9 to 12 IPs per request, all claiming Chrome or Safari | no |
| Perplexity | its own crawler on robots.txt and the homepage only | yes |
Grok’s pattern is the one that breaks user-agent filtering. Across six requests the logs recorded 59 distinct addresses with no reuse, from hosting companies and small ISPs in Brazil, Mexico and Poland. A couple arrived through a network registered to Oxylabs, which sells proxy networks. Every one claimed to be an ordinary browser. A firewall rule keyed on a bot name never sees that traffic, and a per-IP rate limit never trips, because no address was used twice.
Gemini was quieter and nearly as hard to classify. Its fetch came from Google’s own network under a user agent that was literally just “Google”, from an IP on none of Google’s published crawler lists, and it read the path that robots.txt disallowed. A commenter suggested Gemini had answered from Google’s index rather than fetching live. The poster replied that the URL was minted seconds before it was pasted, so nothing could have indexed it. Fetchers with no published identity to check against are becoming a pattern, and OpenAI’s always-on agents browse with no published user agent either.
ChatGPT’s sequence is the detail most likely to mislead someone reading their own logs. On its first visit to the brand-new site it fetched the disallowed page before anything had requested robots.txt, and OAI-SearchBot turned up about 76 seconds later to read the rules. On the next ask, ChatGPT refused the page without sending a request. A single log line from that first minute looks like a deliberate violation when the rules check was simply late, so judge a fetcher on a window of a few minutes rather than one entry.
Claude read robots.txt first and refused a disallowed path outright. One commenter reported the opposite on their own site, with no robots.txt request in their server logs from either the claude.ai chat or Claude Code. The poster’s candidate explanations apply to any test like this: the path may not actually be disallowed for Claude-User or for *, the fetch may come from a different client than the one being tested, or an old copy of robots.txt may be cached. Either way, these behaviours belong to a specific client, not to a brand.
OpenAI and Google both document that fetches a user asks for may skip robots.txt, so neither is doing anything it did not publish. Compliance is not the finding. Identification is: blocking by user agent stops only the fetchers that tell you their name. The test ran for one day with a few runs per assistant, enough to show that disguised fetch traffic exists and not enough to size it. It also measures only fetches a user triggers. Training crawlers and search indexers never find an unlinked page, so nothing here says what GPTBot or ClaudeBot do on your site.
What to do
Run the same test before you trust any AI bot report built on user agents. The setup is small:
- Mint one URL per assistant with a random token in the path, print a unique code on each page, and link them from nowhere. Keep them out of the sitemap and out of internal links.
- Log IP, user agent, path and timestamp at the edge or in your access log.
- Add a second canary under a disallowed path to see which assistants request it anyway.
- Never open the links in your own browser before pasting them. The poster’s first run logged nothing but the poster’s own Brave session, which made every assistant look like the same Windows browser.
- Verify any IP that matters the way Google documents for its own crawlers: a reverse DNS lookup on the IP, then a forward lookup that resolves back to the same address. A commenter suggested running that on the Grok addresses.
# canary paths, linked from nowhere
/_whofetched/chatgpt-7f3a9c
/_whofetched/gemini-2b81de
# second canary, to test the disallowed case
User-agent: *
Disallow: /_whofetched/blocked/Keep user-agent rules for the crawlers that identify themselves honestly, since they still work on exactly those, and resist tightening robots.txt on the strength of one day of logs. A robots.txt rewrite aimed at AI bots took one site out of Google. Anything arriving as Chrome from a Brazilian hosting provider is a network-level problem, and the first step is knowing it is there.