AI visibility vendors leave their own sites open to AI crawlers

A member of r/TechSEO who runs a small publication about AI search and keeps a directory of tools and agencies selling answer engine optimization fetched robots.txt and llms.txt from all 118 domains in it. No AI crawler was blocked at the root by more than one of those sites.

116 of the domains serve a robots.txt at all, and 64 serve an llms.txt that returns 200 and opens with a markdown H1. The audit made one GET per file with a 12 second timeout, parsed robots.txt per RFC 9309, and counted a crawler as blocked only when its own group disallowed / without an allow for /. The 14 agents checked cover the training, search and user-triggered agents of OpenAI, Anthropic and Perplexity, plus Google-Extended and Applebot-Extended, the tokens Google and Apple use for AI opt-out, and CCBot, Bytespider, meta-externalagent and Amazonbot.

Open rules are no surprise for some of those agents. A company whose product is getting cited in AI answers has no reason to disallow OAI-SearchBot or PerplexityBot, because blocking retrieval would break the thing it sells. The training crawlers are a separate decision, and that is where the result says something: the firms advising clients on what AI systems may do with their content have taken no position on their own sites. The thread does not draw this conclusion, but the likelier explanation for a vendor set with almost no root-level stance on any agent is not a policy. It is a default file nobody revisited.

Declared policy is also not access. A robots.txt that allows a crawler says nothing about whether edge bot protection lets it in, and the gap runs in both directions, including the case where bot protection blocked OAI-SearchBot on a site whose robots.txt permitted it. Fetching a vendor’s robots.txt shows what it has decided, not what its server does.

The llms.txt half of the audit is where the pitch and the evidence separate. Just over half the vendor set publishes one. The poster compared that with Ahrefs research across 137,000 domains, where about 28% published the file, and called the vendor rate roughly twice the base rate. A commenter added that in the same research, 97% of sites received no llms.txt request at all, and the poster found no source showing the file changes citations.

The poster and one commenter said the only llms.txt hits they have seen came from SEO crawlers, including LLM-driven audit prompts. The commenter added that there is no way to tell whether those prompts parse the file or only check for a 200, and that none of the major LLM providers explicitly looks for and parses the file right now. Taking that at face value, a 200 on /llms.txt is not evidence that anything reads it. Serving the file is cheap and harmless. Selling it as AI visibility work is the part with nothing behind it.

What to do

• Fetch the vendor’s robots.txt before the pitch call. If AI crawler policy is part of the deliverable, ask which agents their own file disallows and why.
• Grep raw access logs for /llms.txt, then check the requesting user agent and confirm it by reverse DNS. Requests like these never show up in analytics, so the only place to see them is the server log.
• Keep an llms.txt you already publish. Do not report it as a visibility gain, and expect the requests to come from SEO tools.
• Set the root-level policy per bot type deliberately: allow the search and retrieval bots if citations matter, and treat the training crawlers as their own call.