A 200 OK in your logs can mean the AI crawler got an empty page
Summary
A 200 OK in server logs can hide an AI crawler request that received an empty page template, and Moz's new log analysis method finds those requests through the bytes-sent field.
When requesting a log export, ask for bytes sent by name, then sort each AI user-agent by that field and check the smallest responses first.
Moz published a method for reading server logs for AI crawler access, built on a log sample in which three AI agents requested the same pricing page and only one received the full page. The other two logged a timeout and a 200 OK that carried a fraction of the expected bytes.
Every line in an access log records one request, and Moz’s example line holds the fields the analysis runs on:
203.0.113.42 [12/Aug/2026:14:28:19] "GET /guides/pricing" 200 48213 "ChatGPT-User/1.0"
The IP, the timestamp, the URL and method, the response code and the declared user-agent are the parts most people read. The number sitting between the status code and the user-agent is the bytes sent, and it is the field that decides whether the crawler got anything at all.
GPTBot, OpenAI’s training crawler, got a 200 OK for that pricing page and a small fraction of the 48,213 bytes a working request received for the same URL. Moz’s explanation is client-side rendering: when the browser assembles a page from JavaScript instead of receiving ready HTML, a bot can be handed the template with none of the text in it. The status code looks identical either way, and so does any crawl report built on status codes.
The timeout row belongs to ChatGPT-User, the agent that fetches a live page when someone in ChatGPT asks about one. It waited more than 31 seconds on a cache miss and hung up, and the server wrote a 499 with no bytes sent.
Moz sorts AI bots into four tiers by purpose and timeline, beginning with training, and those tiers decide which of these rows is urgent. On that logic, a training crawler receiving a shell costs the site its text in a future model, while ChatGPT-User giving up costs an answer a person is waiting for right now. Both arrive as single lines in the same export, so the user-triggered fetchers are the likelier place to start.
A third failure in the sample has nothing to do with rendering or speed. Claude-User, Anthropic’s user-triggered fetcher, asked for the San Diego location page and got a 403 Forbidden back in six milliseconds. A rejection that fast most likely landed before the request reached the application, which points at the web application firewall or the bot management rules rather than anything in the site’s own configuration, the same pattern as bot protection blocking AI crawlers that robots.txt allows.
Crawler failures of this kind surface nowhere else. Prompt tracking reports what an assistant said about a site and analytics referrals report when a human clicked through, but neither records whether the assistant could fetch the page, which is why chatbot page fetches show up only in server logs.
Where the logs came from decides whether the counts mean anything. When a CDN answers a request from cache, the origin server never sees it, so origin logs undercount requests for exactly the pages that cache well. Moz calls CDN logs the gold standard for that reason.
What to do
- Ask for a sample of a few hundred lines before anyone exports anything larger. It confirms the file opens, that your tools read the format, and that the fields you need are present, rather than finding out after a week spent exporting 40GB.
- Name the fields in the request: client IP, requested URL and method, date and time, user-agent, response code, and bytes sent. Ask for response time and cache status as well. Neither is in nginx’s default log format, but a CDN or load balancer often records both already.
- Pull from the CDN where you can, and treat origin logs as a floor on crawler volume rather than a count.
- Turn on bot verification when importing. The user-agent is self-declared, so anything can call itself GPTBot, and Screaming Frog’s Log File Analyser has a “Verify Bots When Importing Logs” option for it.
- Sort by bytes sent within each AI user-agent and read the bottom of that list first. The near-empty 200s are the findings a status-code report will never produce.
How much data to request depends on the question. Moz gives three starting points:
| Question | Range |
|---|---|
| A specific problem | 1 to 2 weeks |
| General crawling behaviour | 1 to 2 months |
| Training crawler visits | 90 days or more |