ChatGPT clicks go to pages its index crawler already fetched
Summary
ChatGPT referrals mostly land on pages OpenAI's index crawler had already fetched, 92.8% of them in one online shop's server logs.
Check server logs for OAI-SearchBot coverage of the page templates that earn ChatGPT clicks.
The operator of a large online shop matched 7,050 ChatGPT visits to product pages against the shop’s server logs and found that 92.8% of those pages had already been fetched by OAI-SearchBot, OpenAI’s search index crawler, before the click. The operator, who posts as u/DrKJING and keeps the shop anonymous, published the breakdown in a thread on r/TechSEO, drawn from 26 days of logs with bot hits verified against OpenAI’s published IP ranges.
Live fetches were the minority path. Only 6.5% of those visits had a ChatGPT-User request in the hour before the click. ChatGPT-User is the fetcher OpenAI uses when a conversation needs a page right then, and for 33.2% of the pages it never appeared at all. The sample covers Oct 16 to 25 and follows the same shop’s utm outage measurement, with the outage days left out.
The thread breaks the visits down by how long before the click each bot had last fetched the page:
| Last fetch of the page | OAI-SearchBot (index) | ChatGPT-User (live) |
|---|---|---|
| None in 26 days | 7.2% | 33.2% |
| Within 1 hour | 1.5% | 6.5% |
| 1 to 24 hours | 38.8% | 13.6% |
| More than a day | 52.5% | 46.7% |
Index coverage was not confined to the pages ChatGPT happened to link. OAI-SearchBot had fetched 69.8% of the shop’s active products and 96.6% of the products that got Google clicks, so the 92.8% figure largely reflects how much of the live catalogue the bot crawls anyway. The operator’s reading is that ChatGPT answers mostly from its index and reuses earlier fetches, with the caveat that logs cannot show what any single answer was built from.
The better bet, on this evidence, is to treat prior OAI-SearchBot coverage as the thing to protect and live-fetch speed as close to irrelevant. That also splits OpenAI’s crawlers by what they are worth: OAI-SearchBot feeds retrieval and these clicks, while GPTBot, which OpenAI documents as its training crawler, plays no part in retrieval. A robots.txt that blocks everything with OpenAI’s name on it gives up the referrals along with the training data.
Request volume points at a different bot entirely. Meta’s meta-externalagent made 53.6% of the AI bot requests on this shop against OAI-SearchBot’s 15.0%. AI bots together were 1.1% of all requests, so neither the site-wide total nor a league table of crawlers says which bot earns traffic. The join that does is bot hits against referral landings for the same URL, and that is log work: ChatGPT clicks arriving with no tag or referrer show up in analytics as direct traffic.
What to do
ChatGPT sessions in this shop started on a product page 75.5% of the time, against 57.8% for Google organic, so the product template is the first place to measure.
- Pull OAI-SearchBot and ChatGPT-User as separate user agents, verify the hits against OpenAI’s published IP ranges, then compute coverage per template: the share of live URLs in that template with an index fetch in the last month. A site-wide AI bot count will not show a template the index crawler is missing.
- Check that robots.txt, the WAF and any CDN bot rules allow OAI-SearchBot, including where GPTBot is deliberately blocked.
- Treat the indexed copy as days old. More than half of these pages had last been fetched over a day before the visit, so the safer assumption is that price, stock and title changes made since then are not in what ChatGPT reused, and that the facts worth being quoted on belong in the server HTML.