Mueller: blocking one popup JS file in robots.txt is fine

it is fine to use robots.txt to stop…

Summary

Google's John Mueller said a site may use robots.txt to keep an age-verification popup out of Googlebot's rendered page, as long as the blocked file does nothing else.

Find the script that injects the popup and confirm it has no other job before adding the disallow.

Google’s John Mueller said it is fine to use robots.txt to stop Googlebot from rendering an age-verification popup that visitors are required to see, provided the blocked file does only that one job and the page still loads without it. He was answering an SEO on Reddit who described a site pulling its popup from a separate domain through an API, and Search Engine Journal’s report carries both of his replies.

Disallowing crawling on the popup’s own domain “would generally work”, Mueller wrote. A site that uses “a specific JS script file” for the popup “could also disallow that specific javascript file via robots.txt”, he added, with one caveat: he would not do that on a site whose only JS file does a bunch of things.

He came back hours later with both routes in full. When the popup comes from another domain and that domain’s robots.txt disallows crawling, Googlebot will try to fetch the resource while rendering, fail, and have nothing to render the overlay with, which Mueller said is how many ad services already work. When that robots.txt is not yours to influence, the remaining option is to block your own loader on your own site, a file along the lines of load-verification-popup.js. Mueller attached one condition in his own words: “You need to make sure that your pages still load when those URLs are not fetched.” For a popup like this, he said, that is usually not a problem.

What makes the popup case safe is the mapping between one URL and one feature, not anything special about age gates. A dedicated loader is one feature in one file, so withholding it removes one overlay and leaves the article or the product page intact. A single application bundle usually also mounts the navigation and the main content, so disallowing it strips the page down to an empty shell. That is how blocking CSS and JS in robots.txt breaks rendering, and it is the mistake that temporarily cost sites their rankings when blocking JavaScript was common practice. Mueller does not say where the line sits, but the better bet is to treat any file whose name carries a build hash as a bundle until the network panel proves otherwise.

Finding the file that injects the popup takes one pass with the browser’s network panel open: find the request that arrives when the overlay does, and check whether anything else on screen depends on it. Once the disallow is live, Google’s URL Inspection live test reports the resources it could not fetch and shows the rendered HTML, which is where to confirm the body copy survived.

One limit is worth keeping in view. A robots.txt rule binds only the crawlers that read it, Googlebot among them, and a popup injected by JavaScript is already invisible to any bot that never executes the script. ChatGPT, Claude and Perplexity still do not run JavaScript, so this disallow changes nothing for them either way.

What to do

  • Identify the exact file that injects the popup before writing any rule. If it is a hashed build chunk that also renders your content, stop there and ask the vendor or your developers for a standalone loader.
  • Block at the popup provider’s domain when you can, since that keeps your own robots.txt free of rules that future application code might grow into.
  • Write the disallow against the single loader path, not the directory that holds it. A folder-level rule will catch application code the next time someone moves a file.
  • Run the live URL Inspection test afterwards and read the rendered HTML rather than the screenshot. The main content has to be there with the script missing, which is the condition Mueller attached.