Brave sells your pages to AI and robots.txt can't block it

Quoted from Brave Search Crawler help page: The Brave Search crawler does not advertise a
Quote: Brave Search Crawler help page

Summary

Brave's crawler advertises no distinct user agent, so the only robots.txt rule that keeps a site out of the search index Brave licenses to AI companies is one that also blocks Google.

Run a site: search for your domain on search.brave.com to see which pages are in there.

The New York Times sued Brave on 2 November over the search index Brave licenses to AI companies, saying the company copied more than 30 billion webpages “without identifying itself or providing website owners with any mechanism to block its crawler.” Search Engine Journal’s writeup by Slobodan Manic, who publishes the No Hacks blog, points out that Brave documents the second half of that claim itself. Brave had not responded to the complaint when the piece ran.

A crawler with no name

Brave’s crawler help page says the bot “does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them.” It takes its permissions from Googlebot instead: if a page is not crawlable by Googlebot, Brave’s bot will not crawl it either. There is no token to put in a User-agent: line, so the only robots.txt rule that reaches Brave is one that also turns Google away.

Brave points site owners to the robots noindex directive instead, and notes on the same page that robots.txt was never an indexing control. A noindex that drops a page out of Brave drops it out of Google too. For any page that earns its keep in search, that is not a trade worth making, so most sites cannot block this index and can only measure what is in it. Brave also has to re-fetch a page before the delisting applies, and because the crawler carries no name, server logs will not show when that happened.

What Brave sells is not search traffic

Brave sells access to the index through the Brave Search API. Brave’s About page names Cohere and Mistral, which both build large language models, alongside Perplexity and You.com as customers, and the Times’s complaint adds Anthropic and xAI to the list of AI companies it says Brave supplies. None of the five is being sued. The complaint quotes Brave describing one part of the product, “LLM Context,” as handing over “extracted page chunks and source metadata” sized for a model “without need for further scraping.”

An AI company reading a licensed copy never touches the publisher’s server. The user-agent rules practitioners have accumulated for training crawlers and retrieval bots, GPTBot and ClaudeBot on one side, PerplexityBot and OAI-SearchBot on the other, do not apply to this route, and neither do the firewall rules that catch bots robots.txt misses. Even the shared training opt-out Google, Microsoft and OpenAI agreed to honour assumes a crawler that says who it is. Whether the answer built from those chunks sends anyone back to the site is the AI company’s decision, and it is not something Brave is selling.

Manic ran the check on his own site. A site:nohacks.co search on Brave on 3 November returned 16 of his pages on the first results page, and the homepage description in the result was one he had replaced earlier that day, so what Brave serves is a stored copy rather than a live fetch. His robots.txt names 13 AI bots and blocks two of them, Meta’s and ByteDance’s. There is no line he could add for Brave.

What to do

  • Run site: plus your domain on search.brave.com. Everything that comes back is in the index Brave licenses, and the check takes under a minute.
  • Treat noindex as a per-page decision rather than a policy. It is the only exit from Brave’s index and it costs the page its Google listing, so it is worth it only for content whose value is not search traffic, such as paywalled or separately licensed material.
  • Do not count existing AI bot rules as coverage. Robots.txt lines and bot protection stop crawlers that fetch your server; neither touches a copy sold out of someone else’s index.
  • Expect less from the litigation than the headlines suggest. Three more papers, the Daily News, the Chicago Tribune and the Orlando Sentinel, filed a separate case the same day, and News Corp sued in July 2026 arguing Brave hides its crawlers so publishers cannot detect or block them. A ruling on the Times’s pages will not automatically cover anyone else’s.