Google, Microsoft and OpenAI will honour one AI training opt-out
Summary
Google, Microsoft and OpenAI have each said they will read one robots.txt group, `__usage_ai-training`, as a site's AI training preference, and none of them honours it yet.
Write that group now, because no rule already in a robots.txt file counts as a shared training preference.
Google, Microsoft and OpenAI each said on 30 October that they will read a single robots.txt rule group as a site’s AI training preference, per Search Engine Journal’s report. The group is named __usage_ai-training, and it comes from an Internet Draft that Anthropic’s Kevin Kelley submitted to the IETF’s AI Preferences working group the same day. None of the three honours it yet.
The draft adds a name rather than a directive, which matters because Google’s robots.txt parser drops lines it does not recognise. A site writes an ordinary group under that name, using the Allow and Disallow lines it already knows:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example, from the draft, opts the whole site out of training and leaves the press kit in. Two details decide whether a current robots.txt file already says anything: a wildcard group can never match the token, and a file with no __usage_ai-training group states no preference, which Kelley notes is true of every robots.txt published today.
The group also says nothing about crawling. It states how content may be used after it is fetched, so a training crawler that respects it still requests the page. Stopping the request remains a job for per-crawler Disallow lines or for blocking at the edge, which can catch AI crawlers that robots.txt lets through.
The same line, different scope at each company
Google’s post, by John Mueller, says a site that opts out through the shared line without specifying Google-Extended will be treated as if it had disallowed Google-Extended, with support aimed at early 2027. Google-Extended covers two uses in one switch: training future Gemini models, and grounding, meaning content from the search index supplied to the model as it answers in Gemini Apps and on Vertex AI. Google’s crawler documentation describes no way to refuse training while keeping grounding, so a shared training opt-out costs Gemini Apps visibility as well. Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode have their own opt-out in Search Console.
Microsoft went the other way. Microsoft’s post, by Krishna Madhavan, a principal product manager on Bing, adopts the shared line for training and adds a separate AI Grounding preference, set per page and defaulting to allow. A third value, DisallowExtended, keeps a page available for grounding inside Microsoft’s own products, such as Copilot and Bing’s AI Mode, while refusing the grounding services Microsoft sells to other businesses as Web IQ. Microsoft also lets a meta tag or HTTP header set the training preference for a single page, overriding robots.txt there. Either restriction reduces how often a brand appears in AI answers, and neither changes whether the page is listed in Bing search results.
The deadlines are Microsoft’s
Microsoft is the only one of the three replacing a control sites already use, because it is retiring NoArchive and NoCache as training and grounding signals.
| Date | What changes |
|---|---|
| 15 December 2026 | A temporary grounding preference opens in Bing Webmaster Tools, available only to sites still using NoArchive or NoCache that day |
| 1 February 2027 | Microsoft starts honouring the new training and grounding preferences as it crawls |
| 1 March 2027 | NoArchive and NoCache stop counting as training or grounding options |
Pages with no new preference by 1 March default to allowing grounding until Microsoft recrawls them, according to Microsoft’s FAQ. The sharper problem is that the switchover is sitewide: Microsoft says that once it detects either new preference on a site, it disregards any NoArchive or NoCache tags still on that site. Converting a few templates early therefore strips the old protection from every page not yet converted.
OpenAI expects to honour the signal within the next few months and says its current robots.txt policy stays in effect. Neither OpenAI’s post nor its crawler documentation says which rule wins when a shared-line opt-out and a GPTBot rule conflict, so the safer bet is to keep the two consistent until it does. Anthropic had announced nothing as of 30 October, and Kelley said support across its services will take time. Kelley’s draft is an individual submission, not IETF standardization, but three providers committing to read it is reason enough to write the group before any of them ships support.
What to do
- Add a
__usage_ai-traininggroup to robots.txt stating the training preference you want, with Allow lines for the paths you are happy to have trained on. Nothing already in the file counts. - Leave the per-crawler rules in place. GPTBot, ClaudeBot and Google-Extended lines are what govern behaviour until each company ships support.
- Set grounding separately per engine. At Google that means an explicit Google-Extended allow if Gemini Apps answers are worth having. At Microsoft it means the page-level AI Grounding preference, or DisallowExtended if Copilot is welcome but Web IQ is not.
- Inventory the pages carrying NoArchive or NoCache. If any of them rely on those tags to stay out of grounding, record the temporary preference in Bing Webmaster Tools once it opens on 15 December, and convert all of them before setting a single new preference anywhere on the site.