Cloudflare's AI training block now stops Googlebot too
Summary
On Cloudflare, the Training setting now decides whether Google can crawl a site for search. Block and Block on pages with ads shut out Googlebot, Bingbot and Applebot, while Disallow AI Training keeps them for search and refuses training.
Check the Training setting on every zone and the top of the live robots.txt. Sites that welcome some AI companies and block others should turn the sync off and keep writing the file by hand.
Cloudflare’s Bot Preference Sync writes a site’s robots.txt from three dashboard settings: Search, Agent and Training. Cloudflare’s announcement says the generated rules are added above the existing file, so the site’s own directives stay underneath, and that the feature runs on every plan from Free to Enterprise and is on by default for new customers. Search Engine Journal’s piece calls it a good product that still hands a vendor the decision about what a site tells AI crawlers, and does so by default.
The generated rules sit between two comment markers. Cloudflare’s post gives this shortened, anonymized example for a site that allows Search and Agent traffic and disallows Training:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
...
# END Cloudflare Bot Preference SyncCloudflare fills the user-agent list from the bots it tracks and updates it periodically, so the served file can change without anyone at the site editing it.
A Training block now reaches Googlebot
The setting that matters most changed on September 15. In a follow-up post on mixed-use crawlers, Cloudflare made Block and Block on pages with ads apply to crawlers that do search and training under one user agent, including Googlebot, Bingbot and Applebot. A site that sets Training to Block now stops those crawlers from reaching it, search included. Cloudflare points anyone who wants to stop training but keep search to a new option, Disallow AI Training.
Disallow AI Training writes a no-training line into robots.txt instead of blocking at the edge, but only for crawlers Cloudflare labels Accountable. Apple, Google and Microsoft hold that label because they meet, or have committed to meet, four disclosure conditions, including a training opt-out that does not affect search results. For Google, the no-training line is a Disallow for Google-Extended, the robots.txt token Google already offers for keeping content out of Gemini training. Every other training crawler is still blocked outright.
The four Training options now work like this:
| Training setting | Googlebot, Bingbot, Applebot | Training-only crawlers from OpenAI, Anthropic, Meta and Amazon |
|---|---|---|
| Allow | Allowed | Allowed |
| Disallow AI Training | Allowed for search, told not to train | Blocked |
| Block on pages with ads | Blocked on ad pages | Blocked on ad pages |
| Block | Blocked everywhere | Blocked everywhere |
Most existing sites keep the behaviour they had. Cloudflare says earlier Training selections of Block or Block on pages with ads move to Disallow AI Training, and that new ad-supported domains get Disallow AI Training as the recommended default. The risk sits with anyone who opens the dashboard after September 15 and picks Block, reading “Training” as AI companies only. Cloudflare’s AI bot policies documentation adds confusion: it still describes the new-domain default as blocking Training on pages that display ads, with mixed-purpose crawlers covered by every training block. The better bet is to check the setting on any domain added since then rather than trust either page.
One setting per category, not per company
Search Engine Journal’s objection is that the sync sets policy per category, while many sites decide per company. The piece’s author, Slobodan Manic, founder of the website optimisation consultancy No Hacks, allows OpenAI’s GPTBot, Anthropic’s crawler and PerplexityBot, and blocks ByteDance’s Bytespider and Meta’s meta-externalagent, based on what each company gives back. All five companies train models. No Training setting expresses that policy. Disallow AI Training tells OpenAI not to train and, under Cloudflare’s September rules, blocks OpenAI’s and Anthropic’s training crawlers at the edge. Allow leaves nothing in the file that separates Meta and ByteDance from the rest. The sync cannot exclude an individual bot, and Cloudflare’s answer for finer control is to turn it off and maintain robots.txt by hand.
The case for the sync is still real. Cloudflare argues that when robots.txt and edge rules disagree, some crawlers treat the gap as grounds to ignore the site’s preferences, although its post names none of them. Manic’s own robots.txt had welcomed Bytespider by name for months after Manic would have blocked it. For a site with no per-company policy, a file that follows the dashboard beats one nobody has opened in a year. Robots.txt alone still cannot stop a crawler that ignores it, which is why the edge block matters.
Agent traffic gets no Disallow option. Cloudflare says the web has no established robots.txt directive for agents yet, so the Agent choices remain Allow, Block on pages with ads, or Block.
What to do
- Open Security Settings > Configure AI bot policies and read the Training setting on every zone. If it says Block or Block on pages with ads and the site wants search traffic from Google, Bing or Apple, switch it to Disallow AI Training.
- Fetch the live
/robots.txtand look for the# BEGIN Cloudflare Bot Preference Syncmarker. Anything between the markers comes from Cloudflare and can change when Cloudflare updates its bot list, so audit the served file, not the copy in your repository. - If the site treats AI companies differently from one another, turn the sync off, keep writing robots.txt by hand, and put the blocks you intend into WAF (web application firewall) rules so the file and the edge agree.