Google's CEO says agents will browse sites before protocols mature

Google chief executive Sundar Pichai expects AI agents to reach websites two ways: by browsing pages the way a person does, and through protocols written for agents. He called the protocol route “a far more scalable approach” but expects it to take longer to develop. He said it in an interview with Alex Heath, host of the Sources podcast, covered by Search Engine Journal.

Both paths will grow, in Pichai’s account, with “a chaotic evolution” toward interoperable standards along the way. Protocols taking longer to build means the agents that reach pages in the meantime are the browser-driven kind, and Google already names the one it runs. Google-Agent is the user agent its server-side agents use to navigate the web and perform actions at a user’s request, and Google’s fetcher list files it as a user-triggered fetcher.

Google says user-triggered fetchers usually do not follow robots.txt directives, because an end user asked for the fetch. A Disallow line aimed at Google-Agent therefore records an intention rather than enforcing one. Limiting that traffic has to happen where requests are actually refused, at the CDN, the WAF or the origin. That is also where a site that wants Gemini’s agents to work needs to check that nothing is refusing them already.

Identification is where a rule goes wrong. Google warns that user-agent strings can be spoofed and publishes the IP ranges its fetchers operate from, so a rule written against the string alone admits anything that copies the name. Allow and deny decisions belong on the published ranges.

Heath put the commercial problem to Pichai directly. He noted that Google runs an ads business that depends on engagement, and cited Amazon cutting off Meta’s Muse, a personal agent that opens a browser and fills out forms, from shopping on Amazon.com in September. Amazon’s stated reason was that the agent does not identify itself when it browses, and Muse works by reading the page’s accessibility tree in a real browser. Pichai answered that the right answer longer term is to put users first, and that the market will settle into an equilibrium that then shifts. He did not say how Google’s ads work when an agent does the browsing.

For a site owner, the useful part of that exchange is the criterion Amazon used: whether the agent identifies itself. Google-Agent passes it, because it names itself and Google publishes its address ranges. A policy that allows identified, verifiable agent traffic and refuses unidentified browser automation follows from that criterion, and a site can write it now without knowing how any protocol settles.

UCP, the agent commerce standard Google built with a group of companies, is one of the protocols Pichai named, and its reach today is narrow. Merchants adopt it for agentic actions in AI Mode and Gemini, starting with direct buying, while Google’s UCP overview marks lodging bookings and food features as coming soon with a waitlist for each. Pichai also said Gemini’s agent features, currently available mainly to Ultra and Pro subscribers, will open up more widely over the next few months. That puts the near-term volume on the browsing path rather than the protocol one.

What to do

• Find out whether Google-Agent is being refused. Check CDN, WAF and bot-management rules rather than robots.txt, and confirm against Google’s published fetcher IP ranges instead of trusting the user-agent string.
• Split bot rules by type. A block written for training crawlers like GPTBot or ClaudeBot should not also catch a user-triggered fetcher acting for someone who asked Gemini to do something on your site.
• Keep the primary action on each page operable in a browser session, since that is what a hosted agent is driving. Google’s own advice on building pages agents can operate asks for real buttons and layouts that do not move.
• Schedule UCP work on its own timeline. Merchant coverage starts with direct buying, lodging and food are still waitlisted, and Pichai expects the protocol side to arrive slower than the browsing side.