AI-readiness tools check that schema exists, not that it works
Summary
AI-readiness scorers check that schema, llms.txt and bot rules are present rather than whether a crawler can use them, according to an r/TechSEO thread that lists four tests the tools skip.
Before a readiness score goes into a client report, crawl for pages that carry noindex while robots.txt disallows them, since no checker in the thread tests for that conflict.
Most tools that score a site’s AI search readiness confirm that schema, llms.txt and AI bot allowances are present, not that any of them function, according to a thread in r/TechSEO asking whether any checker tests what AI crawlers actually receive. The poster listed four correctness tests they had not found in a tool, and a commenter who disclosed working at Lumar, a technical SEO crawling platform, replied that its free checker covers one of them fully and one in part.
Presence checks are cheap to run, which is why the tools run them. Whether the published artifact does anything is a separate question: llms.txt showed no link to AI citations across 300,000 sites, and facts placed only in schema came back in zero of 12 AI answers.
The Lumar reply covers the two tests that can be answered by reading what a site publishes. It skips the two that depend on a full crawl or on how the server answers a particular request.
| Test | Covered |
|---|---|
| Raw HTML against the rendered DOM | Yes. Lumar’s agentic readiness checker flags content that appears only after JavaScript runs |
| robots.txt rules per bot type | Partly. The checker merges groups per bot token and flags contradictory Allow/Disallow pairs and misspelled tokens, but does not separate training crawlers, search bots and user-triggered fetchers |
| noindex on a page that robots.txt disallows | No. Two commenters called this a job for a conventional crawler such as Screaming Frog |
| A Markdown copy served by user agent at the same URL | No. Only the standards-based form, an Accept: text/markdown request answered with Vary: Accept, is checked |
The uncovered pair carries the larger risk. A URL blocked in robots.txt is never fetched, so the noindex tag on it is never read, and the URL can keep sitting in the index on the strength of inbound links. Finding that conflict means knowing which of a site’s pages carry noindex, which takes a crawl of every URL rather than a read of robots.txt and a sample page. That is why the Lumar commenter pointed at Screaming Frog or a full Lumar crawl rather than the free checker.
Commenters also disagreed about whether Markdown copies are worth serving at all. The Lumar commenter agreed that user-agent-sniffed Markdown is the risky pattern and said the checker does not detect it, while another commenter argued that serving .md copies is wasted effort in the first place.
The likelier conclusion is that these scores are an inventory rather than a diagnosis, and they should be reported to clients as one. The Lumar commenter also said that anything the robots exclusion spec permits, including duplicate groups, empty groups and relative canonicals, is logged as informational and never fails a site, so a passing grade and a clean site are not the same result.
One claim in the thread is worth acting on directly. A commenter reported that some hosts blanket-block AI bots by default, which means the robots.txt a score was computed against may not be the file anyone at the client wrote.
What to do
- Fetch the page with JavaScript disabled and compare it to the rendered DOM. Anything present only in the rendered version is missing for the many AI crawlers that do not execute JavaScript.
- Test robots.txt token by token rather than accepting an “AI bots allowed” verdict: GPTBot and ClaudeBot for training, OAI-SearchBot and PerplexityBot for search retrieval, ChatGPT-User and Claude-User for fetches triggered by a person. A group that resolves to allow for one can resolve to disallow for another.
- Pull robots.txt from the live host, not from the repo, so a host-injected AI block shows up.
- Crawl for pages carrying noindex and cross-reference them against the disallow patterns. Any overlap is a page whose noindex will never be seen.
- If a Markdown variant exists, confirm it is served through content negotiation on the
Acceptheader withVary: Accepton the response, not by sniffing the user agent. The user-agent version serves different content to bots and people at one URL, which the poster noted looks a lot like cloaking.