A $0.35 embedding run finds the off-topic pages in a site crawl
Summary
Running a crawl's own text export through an embedding model scores every page by how far its subject sits from the rest of the site, which surfaces off-topic sections no standard crawl report flags.
Pull Search Console clicks for every page the distances flag, and keep anything earning clicks whatever its radius.
Sitebulb, a desktop crawler used for technical audits, published a guest guide on October 8 that turns an ordinary crawl export into one number for how tightly a domain sticks to a single subject. Sitebulb’s guide was written by Marcel Pękala, an SEO specialist at the Polish e-commerce marketing agency FEB.
Pękala embedded 767 pages of the agency’s own site with Google’s gemini-embedding-2 model for about $0.35 in API usage.
How the score is built
Every page becomes a vector, the average of all those vectors is the site’s centre, and Site Radius is one page’s cosine distance from that centre. Site Focus is the domain-level score, 1 divided by 1 plus the average radius, which came out at 0.85 for Pękala’s site. A page with a large radius is about something the rest of the site is not about.
Pękala reads the domain score in four bands:
| Site Focus | Reading |
|---|---|
| Above 0.90 | Very tightly specialised, a single-topic site |
| 0.85 to 0.90 | Strongly focused, with some peripheral sections |
| 0.75 to 0.85 | Broad but coherent, several related topics |
| Below 0.75 | Diffuse, often sponsored posts, a stale blog, or a merged site |
The metric names are borrowed. Pękala points to the 2024 leak of Google’s Content Warehouse API documentation, which exposed attributes called siteFocusScore and siteRadius, and writes that these two metrics try to approximate the same judgement from outside. The leak named those attributes without describing how Google computes them, so a 0.85 produced by a Gemini model is not Google’s 0.85. The safer use is relative: compare sections of your own site against each other, and watch the number move after you publish or prune.
Where the shortlist came from
The input is a setting configured before the crawl. A Content Extraction rule takes the CSS selector of the main content container (#content on Pękala’s site) with Text as the format, so the export carries what a reader sees rather than navigation and markup. The Website Text report under Bulk Exports then produces one CSV of URL, title, meta description and body text, 850 rows in this run. Cleaning cut that to 767 pages, mostly query-string twins created by a page builder parameter, each dropped only after the script confirmed a clean equivalent existed.
The clearest finding was inside the site’s marketing glossary. Entries defining marketing terms such as organic traffic and SEM sat at a radius of 0.12 to 0.13, close to the core. The glossary’s general IT entries drifted to roughly double that, with TLS furthest at 0.2993, then TLD and neural networks. One section was doing two different jobs, and Pękala notes that no classic report had ever flagged those pages.
Pękala also checked the pages closest to the centroid: four service pages, the homepage and ten blog posts about content marketing, performance marketing and e-commerce campaigns. That is what the agency sells, so the centre landed where the business is. Had the nearest pages not been the ones the site exists for, the far end of the list would have been measuring something else entirely.
The TLS entry was deleted, and its distance is not what justified that. Search Console showed zero clicks against 2,080 impressions at an average position of 9.2, a first-page ranking that nobody clicked. Distance alone would not have carried the decision, and Pękala’s own caveats explain why: author archives, tag pages and pagination orbit far from the centre because of their template rather than their subject, and a page with almost no text embeds its template instead of its content. For those page types the guide’s rule is that a high radius is a noindex or internal-linking question, not a content decision.
What to do
Pękala does not claim the script beats the tools that already do this. Screaming Frog ships a Low Relevance Content report, and free centroid visualisation tools exist. The gain of the script is control over which model does the embedding and which pages enter the centroid.
- Set the Content Extraction rule before crawling. It is a pre-crawl audit setting, and without it the Website Text export has nothing to hand over.
- Dedupe before averaging. Drop parameter twins once a clean version of each is confirmed to exist, drop anything under about 50 characters of text, and report pages with identical content under different paths rather than quietly removing them.
- Read the model’s token limit from the API instead of assuming it. Gemini-embedding-2 takes 8,192 tokens where gemini-embedding-001 takes 2,048, and both truncate an over-long page without raising an error, returning a vector that looks perfectly normal. Only three pages here were actually over the limit, which is how the problem stays invisible.
- Run the Colab notebook that ships with the guide rather than rebuilding the pipeline.
- Treat the per-page distances as a review queue. Pull clicks, impressions and position for every shortlisted URL, and keep anything earning clicks whatever its radius.