OpenAI's search crawler now outpaces its training bot in logs

OpenAI’s search crawler, OAI-SearchBot, now generates more log events than its training crawler, GPTBot, according to a Botify study covered by Search Engine Journal. Chris Long, co-founder of the SEO consultancy Nectiv, analyzed roughly 7 billion OpenAI bot log events from the enterprise client data of Botify, which sells log file analysis and AI bot management software. The data runs from November 2024 through March 2026. Over that span, OpenAI’s overall crawl activity roughly tripled after the GPT-5 launch in August 2025.

The three OpenAI user agents in the study moved in different directions:

User agent	What it does	Change in Botify’s data
OAI-SearchBot	Fetches pages for ChatGPT search answers	About 3.5x more events after August 2025
GPTBot	Collects training data	About 2.9x more events over the same period
ChatGPT-User	Fetches a page when a ChatGPT user asks for it	28% fewer events, December 2025 to March 2026

Before GPT-5, the search and training bots ran at roughly even volumes. After the launch, OAI-SearchBot pulled ahead, at about 1.14 search events for every training event. Long offers two explanations for the ChatGPT-User drop and does not pick one: fewer sessions may be triggering live page fetches, or OpenAI may be answering more often from pages it has already stored.

Why a GPTBot block is not enough

Many sites added a GPTBot disallow to keep their content out of model training. OpenAI’s crawler documentation says each bot is an independent setting. Disallowing GPTBot signals that content should not be used for training. It has no effect on OAI-SearchBot, the bot that now outcrawls it in Botify’s data.

The reverse mistake costs more. OpenAI’s documentation states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. A site that blocked every OpenAI token to be safe has likely removed itself from ChatGPT search without meaning to.

ChatGPT-User is a different case. OpenAI says robots.txt rules may not apply to it, because a user starts each fetch, and that it is not used to decide whether content appears in search. A rule for ChatGPT-User is therefore not the way to control ChatGPT search visibility; the OAI-SearchBot rule is.

How much weight to give the numbers

OpenAI is still small next to Google. In Botify’s most recent 30-day window, OpenAI’s crawlers combined logged about 4% of Googlebot’s volume, up from about 1.38% a year earlier. The growth was also uneven: healthcare sites saw about 740% more OAI-SearchBot activity after launch, while travel sites saw 30%.

The dataset skews toward large enterprise sites, and Botify sells the log analysis software the post promotes. The direction of the trend is the useful part. A small site’s own logs may show different ratios.

What to do

1. Read robots.txt for each OpenAI token separately. A site that wants to stay out of training but appear in ChatGPT search needs two groups:

   User-agent: OAI-SearchBot
   Allow: /
   
   User-agent: GPTBot
   Disallow: /

2. Check the firewall and CDN bot rules as well. OpenAI recommends allowing requests from its published IP ranges for OAI-SearchBot (searchbot.json), and a web application firewall (WAF) rule that blocks OpenAI traffic overrides a robots.txt allow. Some managed WordPress hosts block AI crawlers by default.

3. Split log reports by user agent. A single “OpenAI” line hides the fact that the search bot and the training bot are growing at different rates and serve different purposes.

4. Allow about 24 hours after a robots.txt change before checking ChatGPT search, which is how long OpenAI says its systems take to adjust.