A CDN switch speeds up Googlebot while the cache is still empty
Summary
Google says its crawlers speed up as soon as a new CDN answers fast, and those extra requests miss the empty cache and land on the origin.
Pre-warm the cache with the most-crawled and highest-traffic URLs before moving traffic to the CDN.
Google’s crawlers speed up when a site moves behind a CDN, and the speed-up arrives while the CDN’s cache is still empty. Google’s Search Central blog described that sequence on 11 November, in the fifth episode of its Non-Canonical Cabinet series.
Two things happen at once. Google’s crawl rate follows how the server behaves: “fast and stable responses speed up crawling,” the post says, and responses that get slower pull the rate back down. A CDN answers from an edge server near the requester, so connection and response times drop, and Google reads that as a server with room to spare. But a brand-new CDN starts with what the post calls an “empty (cold) cache”: the first request for a URL at any edge is a miss, and on a miss the edge has to go back to the origin for the response. The post says crawler traffic to the origin spikes almost immediately after the switch.
The extra crawling is not an upside, because crawl capacity is rarely what holds pages out of the index. It is load to survive, and most of the work happens before the cutover.
Google’s fix is to fill the cache before the traffic arrives: programmatically fetch the site’s highest-traffic pages, the pages expected to peak soon, and its most frequently crawled URLs through the CDN at a controlled pace. The post leaves two gaps. It does not say where the most-crawled list comes from; server logs or the Crawl stats report under Search Console’s settings supply it. And until DNS moves, the production hostname still resolves to the origin, so a pre-warm script has to target the CDN’s own endpoint or a staging hostname, or it warms nothing.
Blanket bot protection is the bigger risk. Bot protection applied to all traffic without exemptions meets Googlebot with a CAPTCHA, the crawler fails it, and Google’s post warns that pages begin to drop out of Search as a result. Allow verified bots in both the CDN and the firewall settings, and match Google’s crawlers on its published IP ranges; a user agent string alone proves nothing, since anyone can send one. A separate Search Central post in March 2026 announced a new location for those IP range files, so a rule that still reads them from the old path deserves a check before go-live.
When the origin is at its limit, Google asks for a brake rather than a wall. A temporary 429 (Too Many Requests) or 503 (Service Unavailable) tells the crawler to come back later, and the post caps that window at a few hours, or one to two days at most. A 403 or a 404 says the URL is forbidden or gone. Google published a post in 2023 asking site owners not to use those codes for rate limiting, and serving them at scale during a migration risks dropping URLs you only meant to slow down.
What to do
- Pre-warm before the cutover, fetching the highest-traffic and most-crawled URLs through the CDN endpoint at a controlled pace.
- Add the verified-bot exemption to the WAF before the cutover, not after the first blocked crawl, and confirm the rule reads the current Google IP range files.
- Check what the CDN actually served Googlebot in the first days after the switch. A 200 in the logs can still mean the crawler got nothing usable, and some WAFs log a CAPTCHA page the same way.
- Return 429 or 503 if origin load stays critical, for a few hours or at most two days, and never 403 or 404.
- Google’s post covers Google’s crawlers only. An allow rule built from Google’s ranges does nothing for the AI search bots that bring citations; those need their own entries.