Google says AdSense crawler rules also affect Ad Manager
Summary
Robots.txt rules aimed at Mediapartners-Google are no longer just an AdSense setting. Google's documentation says they affect Ad Manager and other ad products it does not name, and the crawler ignores rules written for all bots.
Sites that block it on purpose should check which ad products that block now touches. Sites that want it kept out need a robots.txt group that names it.
Google has reworded the Mediapartners-Google entry in its special-case crawlers documentation so that the crawler is no longer described as an AdSense-only bot. Search Engine Journal’s piece compared the two versions. The old text said crawling preferences addressed to Mediapartners-Google “affect Google AdSense.” The new text says they “affect various ad-related Google products (for example, Google AdSense and Google Ad Manager).”
Google’s new wording gives examples, not a full list. Search Engine Journal notes that Google does not say whether products such as AdMob, Google’s mobile app ad network, or Google Marketing Platform are covered. Google described the change as a generalization of the documentation. The safer assumption is that a robots.txt block on Mediapartners-Google reaches every Google ad product you run, including ones Google has not named.
The same entry carries a second rule that matters more in practice. Mediapartners-Google ignores the global user agent (*). A User-agent: * group with Disallow: / does not stop it. Only a group that names Mediapartners-Google does. Google’s page explains why: special-case crawlers work under an agreement between the site and the ad product, they “may ignore robots.txt rules,” and so they crawl from a separate IP range published in special-crawlers.json.
What to do
Search your robots.txt for Mediapartners-Google first. If a named group blocks paths where you serve ads through Ad Manager or AdSense, decide whether that block is still intended now that Google ties it to more than one product.
If you want the crawler kept off a path, give it its own group. Rules under * will not apply:
User-agent: Mediapartners-Google
Disallow: /members/When you check its requests in server logs, verify them against the special-case crawler list, not the Googlebot ranges. Google’s page says these crawlers resolve on reverse DNS to rate-limited-proxy-***-***-***-***.google.com. A log filter built only on Googlebot IP ranges will flag them as fake bots.