Google documents how to tell Googlebot when to retry a 503

Google updated its reduce the Google crawl rate documentation on October 6, and the substantive change sits in the emergency section. When a site returns a 503 or 429 to shed crawler load, the guide now tells site owners they can also include a Retry-After header saying when Google’s crawlers may retry, either as a delay in seconds or as an absolute UTC date and time, as defined in RFC 9110. Google added example code for both forms. Barry Schwartz, founder of Search Engine Roundtable, compared the live page with the archived version and found that the rest is mostly content moved around the page.

Nothing about throttling changed. Google’s note on the edit says support for Retry-After “is not new (it was already documented in Temporarily pause or disable a website)”, and that putting it in the crawl rate guide “makes it easier to find and understand when urgently reducing crawler traffic”. Returning 503 or 429 was already the documented way to make Googlebot back off, alongside 500.

The two documents get read at different moments, which is why the duplication earns its place. The pause-or-disable page is for a planned outage. The crawl rate guide is what someone opens while the origin is falling over, and until this week it told them to return errors without mentioning that they could attach a deadline to those errors. During an incident, the version with a retry time is the better one to send: a bare 503 tells Googlebot only that the server is unhappy, while a 503 carrying a retry time tells it when to expect a working page.

Google’s examples show both forms, a delay in seconds:

HTTP/1.1 503 Service Unavailable
Retry-After: 120

or an absolute moment, which has to be written as a GMT date:

HTTP/1.1 503 Service Unavailable
Retry-After: Wed, 21 Oct 2026 07:28:00 GMT

Neither form buys more time than the guide already allowed. Google recommends keeping error responses up for a short period only, a couple of hours up to two days, and warns that going longer may hurt the site’s presence in Google products, including URLs dropping out of the index. The slowdown also applies across the whole hostname rather than just the URLs that returned errors, and Google says crawl rate climbs back on its own once the errors stop. Collateral damage from a blunt crawler rule is a familiar failure: a robots.txt rewrite aimed at AI bots took a site out of Google earlier this year.

What to do

• Make 503 or 429 the emergency response rather than 500. Retry-After is documented for those two status codes, so a 500 sheds load without communicating anything about recovery.
• Set the value to a time the server can meet. The guide does not say which format to prefer, but the absolute form is the safer pick when the end is known, such as a maintenance window; seconds suit the case where the estimate is a guess and the clock should restart on every request.
• Put a hard stop on it. Pick the hour you will turn it off before you turn it on, and make that hours rather than days.
• Expect the whole hostname to slow down. If the load is coming from one expensive endpoint, throttling via error responses is a blunt answer, and a crawl-rate problem confined to one path is better handled by fixing that path.