Only 40% of domains persist across Common Crawl snapshots
Summary
Common Crawl's own model of its crawl series found that only about 40 percent of domains persist across snapshots, so a domain absent from one crawl index has probably not been dropped for good.
Check a site across several Common Crawl indexes before concluding that AI training corpora have never seen it.
Common Crawl has modeled how much of the web its own crawls capture and found that roughly 40 percent of domains form a persistent core, present snapshot after snapshot. The blog post, published on 20 November by Common Crawl’s senior research engineer Michael Paris, comes with an interactive simulation whose sliders are the two parameters the model turns on.
Those parameters are coverage and retention. The preprint behind the post treats a crawl series as repeated draws from an urn: each round samples a fraction of the URL population and replaces a fraction of it. Coverage is the share of the reachable web a single crawl holds, and retention is the share of one crawl that carries over into the next.
The 40 percent came out of measuring those two parameters twice, in two different ways. The standard method compares a pair of crawls and asks how much of one is contained in the other. The preprint adds a second method, the discovery curve, which counts how many distinct domains a sliding window of crawls turns up in total. Both are views of the same process, so on a uniform population the two fits should return the same coverage and retention. On Common Crawl’s 2020 to 2025 data at domain level, and on the German Academic Web at URL level, they did not agree, and the authors treat that disagreement as a measurement rather than an error.
Splitting the population in two makes the disagreement go away. The fix is a core of domains that stay in every crawl, sized at roughly 40 percent, sitting alongside a shell with its own coverage and retention that turns over. Even that leaves a residual, which the authors read as a sign the shell is not uniform either, and they hand the problem to follow-up work.
Most domains, then, are not in the core. That is the part worth carrying into an audit: a domain in the churning shell will appear in one crawl index and not the next, and that is the ordinary case rather than a sign of anything. Common Crawl’s post frames its own modeling problem as separating missing from gone, and stops short of advice for site owners. The conclusion is still hard to avoid. One empty index lookup is not evidence a domain has never been crawled.
Where there is a real gap, it is specific to training data. CCBot gathers the corpus many model builders train on, so a domain the shell keeps dropping is thinner in that corpus. It says nothing about the retrieval bots that fetch pages to answer live queries, such as OAI-SearchBot and PerplexityBot, or about the fetchers that pull a single URL when someone pastes it into a chat. Those run their own crawls on their own schedules. Absence can also have a plainer cause than sampling: AI crawlers that do not execute JavaScript never reach links added in the browser, in which case no amount of re-checking indexes will help.
What to do
- Query several consecutive crawl indexes before drawing any conclusion about a domain. A single miss is what the model predicts for most of the web; a run of misses across crawls is the signal worth acting on.
- Read an absence as a discovery question, not a verdict. The shell is where sampling bites, so the thing to check is whether the pages are linked from somewhere CCBot already visits.
- Keep Common Crawl separate from AI visibility in general when reporting to a client. Presence in the corpus speaks to training data, and the retrieval and user-fetch bots need their own check in server logs.
For the mechanics of checking one site rather than the whole web, Common Crawl’s post points back to its earlier walkthrough on measuring crawled coverage of a single website.