Nearly half of llms.txt files a status check finds are error pages

Only 7.04% of 128,315 EU domains serve an llms.txt file in the form the llmstxt.org proposal asks for, according to a crawl posted to r/TechSEO by the operator of machinewitness.eu, an archive that fetches /llms.txt and robots.txt daily from every domain in the Tranco top million, a ranking of the web’s most popular domains, that has an EU country ending or .eu. More domains answered that path with an HTML page than with a usable text file.

The crawl ran through the week of 10 to 17 October with two independent crawlers, and its totals are published as JSON:

Response at /llms.txt	Domains	Share
Text file whose first line is a Markdown heading	9,035	7.04%
Non-empty text file, any first line	10,002	7.79%
HTML, mostly the site’s own error page	10,138	7.90%

A checker that asks only whether the URL returned 200 reports 20,804 domains, 16.2% of the set, as having an llms.txt. Almost half of what it counts is an HTML page, in most cases the site’s error page served at the wrong status code.

Nothing about llms.txt causes that. A server configured to route unmatched paths to a catch-all HTML response does the same thing at every missing file, and a soft 404 is familiar to anyone who has audited 404 handling. What this crawl adds is the scale of it at one fixed path, across a sample large enough that it is not a handful of misconfigured hosts.

Two assertions are enough to fix the count, and neither costs anything: the response content type is not text/html, and the first line starts with a Markdown heading. The heading is the one thing the llmstxt.org proposal requires, and the gap between the 7.79% serving text and the 7.04% serving the format is the text files that lack it. Presence-only scoring is already the weak spot in this tooling, which tends to check that a feature exists rather than what a crawler receives. A status code is the thinnest version of that mistake, because it is the one field in the response that says nothing about the body.

Format is all the crawl counts, and the post says so: the figures say nothing about whether a file is any good or read by any model. On that question Google’s AI optimization guide is flat: sites do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI features, because Google Search does not use them. Chrome’s Lighthouse points the other way with an experimental Agentic Browsing audit that checks for an llms.txt at the domain root. One commenter in the thread read that audit as a sign Google will start using the file once enough sites have one. Another answered that the Lighthouse team and the Search teams do not coordinate, and that an experimental audit is easy to remove. The likelier explanation is the second. The same commenter notes that some desktop apps and coding tools do look up llms.txt, so the file matters, if at all, to those tools and not to Google Search, and a failing llms.txt check is not a search problem.

What to do

• Treat an HTML response as absent. In any llms.txt check, read the content type and ignore the status code; a 200 that returns text/html is a missing file.
• Read the first line before scoring the file as valid. A non-empty text file that does not begin with a Markdown heading is out of spec, so a byte-count or length check will pass files the proposal would not.
• Fix the soft 404 if a request for a path your site does not have returns 200. That misreporting reaches every file-presence audit run against the site, llms.txt included.
• Do not add the file to chase Google Search visibility. By Google’s own guidance it changes nothing there, so the decision rests on whether agents reading your docs are worth serving.