Google explains how to let Google-Agent past your bot defenses

Quoted from Allow Google-Agent traffic, Google crawling documentation: Each Google-Agent…
Quote: Allow Google-Agent traffic, Google crawling documentation

Summary

Google published a guide on 17 November on letting Google-Agent, the user agent for its agents acting on a person's request, through a CDN, WAF or bot-detection service without dropping other protections.

Check the edge rules for a data-center or AI-agent block that already catches those requests before writing any allow rule.

Google published a technical guide on 17 November on how to let Google-Agent through a content delivery network, web application firewall or bot-detection service. Google-Agent is “the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request,” the document says, and allowing it explicitly “helps prevent legitimate requests from being blocked while preserving your existing security controls.” The user agent is not new: Google added it to its crawler documentation in March, per the crawling docs changelog, and Search Engine Roundtable, which flagged the new page, covered it then.

Three signals identify a request. Every Google-Agent request carries a User-Agent header containing Google-Agent and comes from an IP address in Google’s user-triggered-agents.json file. Some requests are also signed under Google’s Web Bot Auth experiment, which signs agent requests so sites can tell real ones from fakes. Those carry a Signature-Agent header set to g="https://agent.bot.goog" plus Signature and Signature-Input headers labeled g, signed under HTTP Message Signatures (RFC 9421) and checkable against the public keys Google publishes at https://agent.bot.goog/.well-known/http-message-signatures-directory. Google’s suggested rule is to confirm the user agent, confirm the IP, verify the signature when one is present, and skip bot challenges for requests that pass. The order matters: a user-agent string costs nothing to copy, so the IP file is what separates a real agent from anything claiming to be one.

Writing that allow rule is the easy half. Google’s troubleshooting section is mostly a list of ways a correct allow rule loses to the rules around it, starting with priority. The rule has to evaluate before broad cloud or data-center ASN blocks, managed bot rules, or category-level AI agent block policies, any of which can override a verified-bot setting. That last case is the live one on Cloudflare, where personal-assistant agents are now a named category to block in one click.

An allow rule that only skips HTTP blocks still leaves the agent stuck. Google asks that the rule also skip interactive CAPTCHAs and JavaScript challenges, including standalone in-page CAPTCHA widgets that fire on search or form submission, and that it cover actions past the first page load: on-site search, filtering, clicking into detail pages, add-to-cart, booking, and the XHR, GraphQL or api.* endpoints behind them. Because an agent is completing a task rather than fetching a document, a block is likely to land mid-journey, where in logs it looks more like a broken checkout than a bot rule.

Per-IP rate limits are the other thing to plan for. Concurrent requests from many different users share the same egress pool, so a limit sized for one visitor can throttle all of them at once, and Google asks that verified Google-Agent ranges be accounted for or exempted.

Nothing in the guide concerns Googlebot, indexing, or AI training crawlers. Google-Agent is a user-triggered fetcher acting for a person in session, so admitting it is a decision to let an agent transact on the site, not a crawl or visibility setting. Nothing in the doc obliges anyone to allow it, and the safer first move for a site that has not decided is to find out what its edge is doing today rather than to open anything.

What to do

  • Search the CDN, WAF and bot-management config for existing rules that would catch these requests: data-center or cloud ASN blocks, managed bot rulesets, and any AI agent or personal-assistant category policy.
  • Build the verify rule as Google describes it, user agent plus membership in user-triggered-agents.json, plus signature verification when the request is signed. Do not allow on the user-agent string alone.
  • Give the rule higher priority than the broad blocks, and make it skip CAPTCHA and JavaScript challenges rather than only HTTP denials.
  • Extend it to in-journey actions and the API endpoints behind them, then exempt the verified ranges from per-IP rate limits.
  • Confirm load balancers and intermediate proxies pass through the original client IP, the User-Agent header, and the Signature, Signature-Input and Signature-Agent headers. Verification fails on a correct rule when a proxy strips them.
  • Keep the IP rules synced with the current contents of user-triggered-agents.json, which Google updates.