Cloudflare now scores verified bots on robots.txt and 429s

Verified-bot status at Cloudflare now carries an ongoing score. Cloudflare’s post on verified bot accountability, published 16 October, describes a system that watches each verified bot’s traffic and rates how consistently it respects the preferences sites publish, expressed as what Cloudflare calls an explainable trust band.

Two signals feed that band to start. One is whether a bot’s observed traffic stays consistent with the site’s robots.txt over time, the behavior RFC 9309 describes. The other is whether the bot slows down, pauses, or keeps sending at the same pace after an HTTP 429, the status code that tells a client it is making too many requests. Consistent backoff raises trust and repeatedly ignoring the signal lowers it, per Cloudflare’s post, which also says isolated violations may not matter because the system looks for repeated patterns.

The band is a range plus the behaviors that produced it, and Cloudflare is explicit that it is not a permanent judgment about a bot or its operator. Cloudflare frames verification as two questions: are you who you say you are, and can we trust you. Web Bot Auth, a scheme for signing bot requests, and BotBase, Cloudflare’s open catalogue of those identities, answer the first and nothing more. The new score is the answer to the second. Cloudflare’s own illustration of the gap is Perplexity, the AI answer engine, de-listed from the verified list in 2025 after it tried to reach resources on sites that had opted out of scraping and then attempted to get around the blocks when those requests were denied.

The robots.txt half of the score runs without any help from the site. The 429 half does not. Cloudflare’s post does not say this, but a bot can only be seen ignoring a rate-limit signal if the signal was sent, so a site that absorbs crawler load by serving 200s to everything generates no evidence either way about the bots hammering it. Answering a heavy crawler with a hard block has the same effect: a 403 is a refusal rather than a request to slow down, and there is nothing in it for the bot to adapt to. A 429 is the response that gets measured, and it reads more clearly with a Retry-After header, which Google’s crawl docs now describe for 429 and 503 responses.

One line in the post restates existing policy and still deserves repeating. A robots.txt sets the minimum of what bots should not do, and nothing about what is permitted follows from what it leaves unsaid; Cloudflare’s example is that a file does not have to spell out that a bot must stop when the site signals that further access is unwelcome by other means.

Coverage is narrower than the verified list might suggest, because the system scores bots that identified themselves. Traffic arriving dressed as a browser sits outside it, and that traffic is not hypothetical: one unlinked test page saw live fetches from 59 IP addresses claiming to be Chrome or Safari. Bot Management still handles that category. Trust bands will not.

Nothing is configurable today. Cloudflare says it is running the system internally to build up history and refine how the observations turn into guidance, with publication on Radar and a customer-chosen trust threshold both described as future steps. Until the thresholds are exposed, the practical value is knowing that an allowlist entry can now decay on evidence.

What to do

• Shed crawler load with a 429 and a Retry-After value, not with a silent 200 or a 403. The first is the only one of the three that the trust band can read, and it is also the one that tells a well-behaved crawler what to do next.
• Read your robots.txt as a floor rather than a contract. Paths it never mentions are not thereby allowed, and a bot that stays inside the allowed paths can still cost the origin plenty.
• Hold off on rebuilding verified-bot rules around trust levels until the scores land on Radar. Guessing which crawlers currently score well is a worse input than the live numbers will be.

Watch out for

A single bad crawl reported to Cloudflare will not get a bot delisted. The system is built on sustained patterns, so one burst of requests through a disallowed path will not move a band. The likelier value of the band is a record of which bots you rely on have backed off when asked, and it will take months of observation before it says much at all.