Scaled content may lose crawling after Google tests a sample

Scaled content often climbs fast and then falls. In a Search Engine Journal piece published 2026-05-07, SEO consultant Dan Taylor blames the fall on a quality check rather than on AI detection. Taylor argues that a large batch of new URLs first gets a freshness boost. Google then tests a representative sample of the batch, possibly grouped by a URL pattern such as a subfolder, and watches how users engage with it. If the sample falls short of Google’s quality bar, Taylor writes, crawl resources are withdrawn and the rest of the batch struggles to gain traction.

Taylor’s example is a brand tracked by Martin Sean Fennon, an SEO who has followed its AI-powered content scaling for months. In Fennon’s LinkedIn post, Ahrefs data shows the brand’s keywords in positions 1 to 3 tripled between October and January. By April a third of those gains were gone, and the decline continued into May. The brand’s presence in Google AI Overviews fell 66% in the last reporting period.

Taylor’s own chart makes the case that AI is not the cause. A brand he launched in January 2021, with no AI content, shows the same early lift fading after a few months. The pattern comes from publishing lots of new URLs at once, whatever wrote them. AI just makes that volume cheap, and volume exposes weak topic selection, editing, internal linking and distribution.

What Google documents and what it does not

The link between quality and crawling has Google documentation behind it. Google’s crawl budget guide says Googlebot’s crawl demand depends partly on page quality and relevance. It lists raising content quality as one of two ways to earn more crawl budget. Adding server capacity is the other.

The sampling step is not documented. It is Taylor’s explanation for the pattern, not something his piece tests, so treat it as a plausible model rather than confirmed behavior.

The bar also moves. Taylor cites a LinkedIn post by Adam Gent, an SEO who says a page quality score Google calls Q* surfaced in the DOJ antitrust trial and changes daily. On Gent’s account, a page indexed six months ago can fall below the bar without anyone touching it, and the first sign is in crawl data: lower-quality pages get crawled less and less, then not at all. Taylor adds that the threshold varies by topic.

Taylor points to Google’s own recent description of the bar. At Search Central Live in Toronto on April 21, Google’s Search Liaison Danny Sullivan asked for unique, specific, authentic “non-commodity” content, per SE Roundtable’s coverage. His running store example contrasts “Top 10 Things to Consider When Buying Running Shoes” with an analysis of why one customer’s shoes collapsed after 400 miles. The likelier failures under Taylor’s model are scaled pages that read like the first example.

What to do

• Wait out the freshness boost before judging a batch. First-month traffic says little about whether the pages will stay. Taylor notes that submitting a URL through Search Console for indexing gives a similar boost, so manual indexing requests inflate early numbers further.
• Track crawl frequency by section. If Taylor’s model holds, the earliest sign a batch failed is Googlebot returning to its subfolder less often. Group server log hits by directory and compare the weeks after launch with the weeks after the lift fades. Apply the same check to older sections, following Gent’s point that pages can lose crawl frequency without changing.
• Raise the floor, not the average. The better bet is to edit every page in a batch to the same standard. Under Taylor’s model Google chooses which pages it tests, so a publisher cannot count on the strong pages being the ones sampled.
• Cut the URLs Google should not spend time on. Google’s crawl budget guide recommends consolidating duplicates, blocking low-value URL variants in robots.txt, and returning 404 or 410 for pages removed for good. A smaller, cleaner inventory leaves more crawling for the pages that matter.