> ## Content Index
> Fetch the complete content index at: https://www.reputation-insider.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Google tells websites how to let AI agents past bot defenses
- URL: https://www.reputation-insider.com/google-tells-websites-how-to-let-ai-agents-past-bot-defenses/
- Published: 2026-11-17T18:30:00.000Z
- Updated: 2026-11-17T18:29:59.000Z
- Description: Google has published instructions for verifying requests from its AI agents and allowing them through website security systems that can interrupt browsing and transactions. The guidance also introduces cryptographic authentication, which remains experimental.
- Author: Marielle Vast
- Tags: Watchlist, #watchlist-main, #hide-recent

Google has published new technical guidance explaining how website operators can identify and permit requests from Google-Agent, the user-directed AI agent that navigates websites and performs actions on behalf of users. The [documentation addresses restrictions imposed by content delivery networks, web application firewalls and automated bot-detection systems](https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent?ref=reputation-insider.com) that may prevent legitimate agent activity even when a website is otherwise accessible.

The instructions cover verification of Google-Agent requests, configuration of security exceptions and troubleshooting of blocked browsing actions. Google is also testing cryptographic authentication that would allow websites to establish the identity of participating agents through signed HTTP requests. The guidance gives publishers a technical method for permitting automated interactions while retaining controls over unauthorized traffic.

01 / 03

## Google defines how websites should verify its AI agents

Google-Agent is used by AI agents hosted on Google's infrastructure to navigate the web and perform actions upon user request. Google introduced the agent earlier as part of its user-triggered fetching infrastructure, making the new publication an operational guide for an existing system rather than the announcement of a separate crawler.

Under the new instructions, each Google-Agent request includes an HTTP User-Agent header containing the identifier **Google-Agent**. The request must also originate from an IP address included in Google's published [user-triggered-agents.json address ranges](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json?ref=reputation-insider.com). Administrators can use these attributes to distinguish requests associated with Google's infrastructure from traffic claiming the same identity without supporting network evidence.

The distinction matters because User-Agent headers can be copied by unrelated automated clients. Google's [documentation on user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers?ref=reputation-insider.com) explicitly warns that these identifiers can be spoofed. A firewall exception based only on a request header could therefore admit traffic that has not been authenticated as originating from Google.

Google recommends that website operators use verification features provided by their CDN or security vendor where available. Otherwise, administrators can implement their own rules using the published IP ranges and, for eligible requests, cryptographic signature verification. Once a request has been verified, Google recommends allowing it to bypass automated challenges that would otherwise interrupt its activity.

Google-Agent belongs to the category of user-triggered fetchers, which generally operate in response to user actions and ignore robots.txt restrictions. That differs from conventional search crawling, where robots.txt provides established controls over crawler access. The new guide concerns security filtering at the network and application level and does not announce a change to Google's search indexing rules.

02 / 03

## Firewall exceptions must cover actions beyond loading a page

Google's troubleshooting instructions identify several reasons legitimate agent requests can remain blocked after a website has configured an allow rule. A common problem is rule priority: broad restrictions on cloud infrastructure, data center networks or AI agents can override exceptions intended for verified Google-Agent traffic. Google advises operators to ensure that the relevant permission is evaluated before those broader blocking policies.

Interactive challenges create another obstacle. CAPTCHA checks and JavaScript verification procedures can interrupt automated navigation even when the underlying HTTP request has been accepted. Google recommends that verified agent traffic bypass these challenges, including CAPTCHA widgets that appear during searches or form submissions within a website.

The guidance also addresses transactions and multi-step browsing. Allowing an agent to load a homepage does not necessarily permit it to search a product catalog, apply filters, open detail pages or complete a booking process. Those interactions may depend on separate application endpoints that are subject to stricter security rules.

Google specifically identifies add-to-cart operations, booking functions, XHR requests and GraphQL or API endpoints as areas that administrators should examine. A website may appear accessible to an agent during initial navigation while refusing the subsequent requests needed to complete a user-directed task. Google recommends checking that verified traffic can reach the relevant endpoints throughout the interaction.

Rate limiting presents a related problem because requests associated with multiple users may originate from the same Google-Agent egress IP address. Security systems applying conventional per-IP thresholds could interpret legitimate concurrent activity as excessive automated traffic. Google recommends that operators account for verified agent address ranges when configuring those limits.

The document also warns that IP allowlists need to remain synchronized with Google's published ranges. Load balancers and intermediate proxies must preserve the original client address and relevant authentication headers so that downstream security systems can verify requests correctly. These requirements make agent access dependent on the configuration of the website's supporting infrastructure as well as its application itself.

03 / 03

## Cryptographic verification remains experimental as agent access expands

Alongside conventional identification methods, Google is experimenting with Web Bot Auth, a cryptographic protocol intended to help websites authenticate automated clients. The technology uses signed HTTP requests to establish that participating traffic originates from an agent holding the appropriate credentials, reducing dependence on identifiers that can be impersonated.

Under Google's [experimental Web Bot Auth implementation](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth?ref=reputation-insider.com), participating Google-Agent requests identify themselves as **https://agent.bot.goog**. They include a Signature-Agent header and associated Signature and Signature-Input headers. Website operators can retrieve Google's public verification keys and validate the signatures using the HTTP Message Signatures standard, RFC 9421.

Google has emphasized that the protocol is still being developed through the Internet Engineering Task Force and has yet to become a universal authentication mechanism. Only a subset of Google-Agent requests currently carries cryptographic signatures, and the company continues to recommend established IP-based verification for requests that do not participate in the experiment.

The distinction limits how website operators can deploy the technology today. Requiring a valid cryptographic signature for every Google-Agent request could exclude legitimate traffic that Google has not yet enabled for Web Bot Auth. Supporting the protocol therefore requires handling signed requests when available while maintaining another means of verifying the remaining traffic.

For publishers and online businesses, the guidance brings automated user-directed browsing into routine website access management. Security rules originally configured to exclude unknown bots can also prevent verified AI agents from completing tasks that users have requested. Operators that want to support these interactions must decide which functions their agents can access and ensure the corresponding security exceptions are implemented consistently.

Google's documentation does not require websites to admit Google-Agent or establish that permitting it will improve search visibility. It provides a configuration framework for operators choosing to support agent activity, while leaving access decisions and associated security responsibilities with the website. The broader deployment of signed requests could eventually make agent identification more reliable, but Google has not announced a timetable for universal adoption.