Meta's Oct 30 terms let it train AI on any business user's site
Summary
From 30 October, Meta's Commercial Terms give it permission to crawl and train on the website of any business that uses its products, including businesses that have never bought an ad.
Before then, set a separate robots.txt rule for each Meta crawler, since the training crawler and the Meta AI search crawler are different agents.
Meta’s updated Commercial Terms take effect on 30 October and give the company permission to crawl the websites of businesses that use its products, and to train its AI products on what it collects, according to Search Engine Journal. The trigger is any business use, from buying ads to running a Facebook Page.
The current Commercial Terms, dated 6 March 2024, contain no crawling clause. That right has lived in the ad terms, where the Self-Serve Ad Terms of 22 August 2025 tie it to placing an ad order and list “research and training” among the permitted uses. The October rewrite drops the ad order, so brands and clients that have a Page or sit in Business Suite without ever buying media are covered too. Continuing to use Meta for business after the 30th is how a business accepts the terms, and the only exit the terms name is to stop using Meta’s products for business purposes.
The clause covers more than the business’s own domain. It reaches domains and pages associated with a business’s products or services, plus URLs made available through the business’s use of Meta’s products or through users’ use of them, and it lets Meta or anyone acting for it run scrapers and bots against those URLs. It also lets Meta “create test credentials” to reach the pages. James Grimmelmann, a law professor at Cornell who reviewed the terms for Nieman Lab, Harvard’s journalism research site that first reported the change on 15 October, said test credentials could technically let Meta sign up as a fake user to get past a registration wall. Meta declined to say whether they would be used on paywalled content.
Meta disputes the reading. Andy Stone, a Meta spokesperson, replied on X on 16 October that a summary of the Nieman Lab report was false, saying the change is there so Meta can pull accurate information about products for sale on third-party sites when it recommends ads and content, and that it is “not about scraping news sites.” Meta’s own crawler documentation assigns that job to a separate agent, Meta-ExternalAds, which it says crawls to improve advertising and other business products. If Stone’s account of the purpose holds, blocking the training crawler would not interfere with the product-data crawling Stone describes.
Meta’s position, in Stone’s post and in its statement to Nieman Lab, is that content owners can still manage how their content is accessed using robots.txt. Neither updated document mentions robots.txt. The lawyers who read the terms for Nieman Lab took the new language as an opening for Meta to argue that its commercial terms override other blocking instructions, and nothing in the terms says whether a business’s robots.txt settings limit what it has agreed to share. Matt Topic, a media and intellectual property attorney representing The Intercept in its copyright case against OpenAI, told Nieman Lab there is no assurance a court would enforce the provision but that publishers should assume one will. The safer read is that robots.txt still governs what Meta’s crawlers fetch and nothing more. It is not a reservation of rights against a contract accepted by continuing to use Business Suite.
Which Meta agent you block decides what you lose. The documentation lists five with different jobs, and two of them say they may ignore the file.
| User agent | Documented job | Robots.txt |
|---|---|---|
| Meta-ExternalAgent | Trains foundation AI models, indexes content | No bypass documented |
| Meta-WebIndexer | Improves Meta AI search results and citations | No bypass documented |
| Meta-ExternalAds | Crawls for advertising and business products | No bypass documented |
| Meta-ExternalFetcher | Fetches single links at a user’s request | May bypass |
| FacebookExternalHit | Builds link previews for shared URLs | May bypass on integrity checks |
What to do
Treat training and citation as separate decisions. Meta says allowing Meta-WebIndexer helps it cite and link your content in Meta AI responses, so a publisher who wants that traffic blocks the training crawler and leaves the search crawler alone:
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-WebIndexer
Allow: /Push that live before 30 October rather than on the day, because Meta’s documentation says robots.txt changes can take up to 24 hours to take effect. The contract question is separate from the file. Anyone who needs an answer on it should read the clause itself, since Meta’s Business Help Center notice described the document changes as reducing overlap and said nothing about crawling or training.
Watch out for
Two Meta agents are documented as able to ignore the file you just edited. Meta-ExternalFetcher may bypass robots.txt when it fetches a page at a user’s request, and FacebookExternalHit may bypass it during security or integrity checks, so a Disallow line stops neither. Fetchers like these do not announce themselves in logs: a canary-URL test found Grok’s fetcher hitting one unlinked page from 59 IPs posing as Chrome or Safari, and stopping that kind of traffic takes edge rules or server-side blocks. Cloudflare now scores verified bots on robots.txt compliance, which is the kind of layer that measures obedience rather than assuming it.
Gated content is the other gap. Robots.txt says nothing about what sits behind a login, which is what makes the test credentials clause worth reading closely if any of your product or editorial pages require registration.