OpenAI crawl activity tripled after GPT-5, led by search bot
Summary
OpenAI's crawl activity tripled after GPT-5 launch in August 2025, with OAI-SearchBot now outpacing GPTBot. OAI-SearchBot and GPTBot have separate robots.txt directives, so blocking one doesn't block the other, sites blocking only GPTBot still get crawled for ChatGPT search results. Review your robots.txt for all three OpenAI user agents and check server logs for load impacts.
What happened
OpenAI’s automated crawl activity roughly tripled after the August 2025 launch of GPT-5, according to an analysis published by Botify and guest author Chris Long, co-founder of SEO consultancy Nectiv. Long analyzed approximately 7 billion OpenAI bot log events from Botify’s enterprise client dataset, spanning November 2024 through March 2026.
OAI-SearchBot, which retrieves content when ChatGPT performs web searches, recorded about 3.5x more events after August 2025. That works out to roughly 2.2 billion additional events. GPTBot, the training data crawler, saw about 2.9x more events over the same period, adding another 1.8 billion events.
The third user agent, ChatGPT-User, moved in the opposite direction. Long reports a 28% drop in ChatGPT-User log events between December 2025 and March 2026. ChatGPT-User fires when a ChatGPT session fetches a page on behalf of a logged-in user. Long offers two possible explanations: fewer sessions may be triggering real-time fetches, or OpenAI may be relying more on stored or indexed resources.
Why it matters
OAI-SearchBot now generates more log events than GPTBot in Botify’s dataset. Before GPT-5, the two bots ran at roughly even volumes, with a ratio of about 0.95 search events per training event. After GPT-5, that ratio rose to about 1.14.
The shift matters for robots.txt decisions. OpenAI’s own documentation confirms that each bot’s robots.txt directive is independent. Sites blocking only GPTBot are not blocking the bot OpenAI uses to surface pages in ChatGPT search answers. Conversely, sites blocking OAI-SearchBot may be excluding themselves from ChatGPT search results entirely, though they can still appear as navigational links.
The post-GPT-5 increases varied by industry. Healthcare sites saw about 740% more OAI-SearchBot activity. Media and publishing saw 702%. Marketplaces, software, and retail ranged from 190–216%. Travel sites had the smallest rise at 30%.
Long also found that the balance between search and training crawls differs by vertical. Media and publishing showed the largest gap favoring OAI-SearchBot (+256% over GPTBot). Healthcare and retail leaned toward GPTBot. Botify and Long suggest OpenAI routes different prompt types to different crawlers: news queries trigger live search, while health and product queries draw more on trained knowledge.
Even after tripling, OpenAI’s crawl volume is small compared to Google’s. In Botify’s most recent 30-day window, Googlebot registered 18.2 billion events versus 887 million from all OpenAI crawlers combined. That puts OpenAI at about 4% of Google’s crawl volume, up from 1.38% a year earlier. Bingbot registered about 5.49 billion events, making OpenAI roughly 14% of Bing’s volume.
What to do
Review your robots.txt for all three OpenAI user agents separately. If you want to appear in ChatGPT search answers, make sure OAI-SearchBot is allowed. Blocking GPTBot alone does not block ChatGPT search crawling. OpenAI’s documentation states that robots.txt changes take about 24 hours to take effect in their systems.
Check your server logs for OAI-SearchBot and GPTBot activity. Compare volumes before and after August 2025 to see whether the tripling pattern holds for your site. If you’re seeing large increases, verify your server can handle the additional load.
Segment bot activity by section or content type. The industry-level data suggests OpenAI’s crawlers focus differently depending on content. Understanding which sections attract OAI-SearchBot versus GPTBot can inform your robots.txt strategy at the directory level.
Factor in the dataset’s limitations. Botify’s data skews toward large enterprise sites. Smaller sites may see different patterns. Botify also sells log file analysis and AI bot management software, and the original post promotes a webinar and product demo.
Watch out for
Blocking the wrong bot. Many sites added GPTBot blocks when it launched, assuming it covered all OpenAI crawling. OAI-SearchBot is a separate user agent with a separate robots.txt directive. If you blocked GPTBot but never addressed OAI-SearchBot, your training data is protected but your ChatGPT search visibility is unaffected. The reverse is also true: blocking OAI-SearchBot while allowing GPTBot means your content can still be used for model training.
Misreading ChatGPT-User drops as declining ChatGPT usage. The 28% decline in ChatGPT-User events measures logged user-initiated fetches, not overall ChatGPT traffic or interest. OpenAI may simply be caching or pre-indexing content more aggressively, reducing the need for real-time fetches.