Google's robots.txt parser silently drops lines it can't read

Google’s robots.txt parser recognizes four rules, user-agent, allow, disallow and sitemap, and silently throws away every line it cannot read. Google’s Gary Illyes and Cherry Prommawin said so in the second video of their Non-Canonical Cabinet series, which Search Engine Roundtable wrote up on October 21.

Nothing warns the site owner when a line gets dropped. In Illyes’s example, a site writes “don’t crawl secret page”, Google reads “don’t crawl”, finds no valid rule, ignores the line and may crawl the secret page anyway. The file still looks fine, because a line that reads as an instruction to a human is invisible to a parser that knows four commands.

Homemade AI opt-out lines fall in that same bucket. A directive a site invents for itself is not part of RFC 9309, the standard Google follows, so it is discarded along with everything else the parser cannot read, and the crawler it was aimed at keeps crawling. The only lever robots.txt gives over an AI crawler is a disallow under that crawler’s recognized user-agent token, which is one of the video’s stated takeaways.

Tokens are per bot, not per company, and the bots do different jobs. OpenAI’s crawler documentation lists GPTBot for training crawls, OAI-SearchBot for ChatGPT search, and ChatGPT-User for fetching a page when someone asks about it, so a disallow on one says nothing about the other two. A fetch test across well-known sites found nine of them blocking GPTBot while leaving OpenAI’s search bot free. The likelier explanation is that the split is deliberate, since the search bot can send referral traffic. Deciding token by token before publishing the file beats discovering the split later in server logs.

Blocking the crawl is also not the same as forbidding training, and Illyes and Prommawin separate the two: keeping an AI crawler off a page differs from telling a company not to train on content it has already obtained. Robots.txt only works forward. Material collected before the disallow went up stays collected, and anything a site wants governed after that point belongs in its terms or a licensing agreement.

Where correct-looking rules break

Non-English characters are the second trap. User-agent tokens hold only letters and dashes, Illyes said, so an emoji can never be a valid token, because an emoji is a multibyte UTF-8 character and no encoding of it meets that requirement. Rule paths can carry those characters, and the video’s advice is to percent-encode them. Google’s parser translates raw characters on its own, but the video warns that other or older crawlers might get confused, so an unencoded rule can hold for Googlebot and fail elsewhere. Encoding matters outside robots.txt too: Google indexes the raw and percent-encoded forms of a bracket query string as separate URLs.

What to do

Four passes over the file cover most of this:

• Read every line and check whether its first token is user-agent, allow, disallow or sitemap, or whether the line starts with a hash. Delete anything else. Crawlers skip comment lines; invented directives give a false sense that something is blocked.
• Name each AI crawler by its published token, one group per bot, and percent-encode any non-ASCII path. For example:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /caf%C3%A9/

• Test the actual URL patterns against the rules instead of reading the file and judging it by eye. That is the first point Search Engine Roundtable pulled from the video, and it is the only way the silent discards surface.
• Keep anything sensitive out of this decision entirely. Robots.txt asks compliant crawlers to stay away; it does not stop an unauthorized client. A path that must not be fetched needs authentication or a server-level block.