A new site's first AI visitor was a live fetcher, not a crawler
Summary
On a brand-new domain, AI assistants fetched pages on a user's behalf before any AI training crawler showed up, and OpenAI's crawler then read the whole sitemap in a single burst. All of that AI reading sent nobody to the site, and GA4 would have shown none of it on either side.
Pull raw server logs now to get a baseline of which AI bots reach your site, verify the biggest rows against published IP ranges, and keep the sitemap and error rate clean through launch week. Read your terms, privacy and docs pages as answers, because those are what a live fetcher pulls when someone asks an assistant to vet you.
The first AI request to reach a site launched on October 7 came from Claude-User, the fetcher Anthropic uses when someone asks Claude a question that needs a live page. It arrived on day 2, an hour before ClaudeBot, Anthropic’s crawler, made its first request. The ordering comes from a log breakdown posted to r/TechSEO by the site’s owner, who forwarded every bot request to their analytics and counted human visits separately in JavaScript.
Over the first five days the site logged roughly 750 bot requests against 207 human pageviews. The AI and Google crawler rows, combining the thread with the fuller table on the owner’s blog, came out like this:
| Bot | Requests |
|---|---|
| GoogleOther | 69 |
| GPTBot | 67 |
| Bytespider | 10 |
| ChatGPT-User | 9 |
| ClaudeBot | 3 |
| Claude-User | 2 |
| meta-externalagent | 2 |
| PerplexityBot | 1 |
GPTBot’s timing is the part worth planning around. OpenAI’s training crawler did nothing for four days, then requested 67 distinct URLs on day 5, one request each, which the owner describes as the whole sitemap in one pass. One domain cannot establish that every new site gets a single early sweep, but if that is the pattern, whatever state the sitemap and the server are in during that window is what gets read, and being ready for it costs nothing.
The live fetches look like a buyer doing due diligence. ChatGPT-User made nine fetches across the five days, and within one minute on day 4 it opened the homepage, docs, API docs, terms, privacy and blog. The owner’s guess is that someone asked ChatGPT to vet the product, and the page set supports it, because that is the run a person makes before signing up rather than an order any crawler would follow. Terms and privacy pages written to survive a lawyer, not to answer a question, now get read on a prospect’s behalf.
None of it produced a visit. The thread reports zero clicks from ChatGPT, Claude, Perplexity or Gemini over the same five days. GA4 would have shown almost none of either side, since it filters bots and AI referrals frequently arrive with no referrer at all.
Every row in that table records what the request claimed to be, not a verified identity. The owner says so directly: classification is by user agent only, with no check against published IP ranges yet. Both kinds of error apply here. Anything can send GPTBot in a header and inflate a row, and real AI traffic routinely does not announce itself, as in the canary test where Grok fetched an unlinked page from 59 IPs while sending Chrome and Safari user agents. OpenAI’s standing-query agents browse with no published user agent, which makes any user-agent tally a floor and not a total.
The measurement also comes from the person who builds PageLens, the analytics tool that produced it, disclosed in the thread alongside the full breakdown on the tool’s blog. One five-day-old domain with one sitemap is a data point, not a benchmark, and the one commenter on the thread called it self-promotion.
What to do
The useful response is to get your own baseline on record before you change anything about how AI bots reach the site.
- Pull the last 30 days of raw server logs and group hits by user agent, then verify the big AI rows by reverse DNS or published IP range. A row you have not verified tells you what something claimed to be.
- Split the AI rows into training crawlers, retrieval bots and user-triggered fetchers. A robots.txt rule naming
GPTBotleavesChatGPT-UserandOAI-SearchBotuntouched, so a block that looks effective in one column can be doing nothing in the others. - Read your terms, privacy and docs pages as answers rather than policies. Those were the pages the live fetcher pulled, and they are what an assistant summarizes back to someone deciding whether to buy.
- Keep the sitemap and error rate clean through launch week specifically. The evidence for a single early GPTBot pass is thin, but the cost of being wrong about it is one bad crawl of a brand-new domain.
Judging any AI crawler rule without a before is guesswork. Right now the comparison is free to collect.