Google crawled a news site less as its soft 404 count climbed
Summary
Google crawled a news publisher's French site far less as soft 404 errors kept multiplying across its country sites, a Search Engine Land case study found.
Check the Soft 404 row in Search Console's Pages report over time and look into a count that keeps rising even when the total is small.
The Brazilian edition of a cryptocurrency news site lost about 90% of its sessions and pageviews in the year after it moved domains. A Search Engine Land case study by Mikael Araújo traces the damage past that migration to indexing problems across all 13 of the publisher’s country sites, soft 404s among them.
The events ran from 2021 to 2023, and the site is anonymized in the piece as xx.com.
The migration started it
In January 2022 the Brazilian site moved from xx.com.br to br.xx.com. Before the move, Search Console showed 15,000 to 25,000 clicks a day. After it, clicks settled at 2,000 to 4,000 a day and stayed there for more than a year. Araújo’s piece says the redirects and other migration steps were never fully carried out, so Google kept crawling the old domain long after the move and split its crawling between the two.
Fixing the migration exposed a bigger backlog. By January 2023, Search Console’s indexing report for the Brazilian site showed these counts:
| Search Console status | Pages |
|---|---|
| Crawled - currently not indexed | 513,369 |
| Alternate page with proper canonical tag | 2,532 |
| Soft 404 | 1,193 |
| Discovered - currently not indexed | 524 |
The team found the site was automatically generating currency converter pages at scale, with URLs like /usd-to-thor?amount=250. The piece describes these thin pages as using up Google’s crawling and pulling down its view of the whole domain.
Why a small soft 404 count still mattered
A soft 404 is a URL that returns HTTP 200 while Google judges it to have no real content. Google’s help page on 404 errors names two common causes. One is returning any code other than 404 or 410 for a page that does not exist. The other is redirecting missing pages to another page, such as the homepage. Pages that exist can be flagged too, as with empty category pages that Google reports as soft 404s.
On the Brazilian site, the soft 404 count was tiny next to the half-million unindexed pages. Araújo describes it as “growing rapidly,” though, and the other domains were further along. The main .com site had 90,400 affected pages, and the Spanish site had 17,700.
The French site shows what came with the rise. Soft 404s started to pile up there in October 2022. Before that, Google made 60,000 to 70,000 crawl requests a day to the French site. Over the following months it made 20,000 to 30,000. Araújo calls this a direct correlation. The piece shows the two trends moving together, but it does not prove that one caused the other.
Either way, the rising count sat in Search Console from October 2022. The fix did not start until Jan. 31, 2023. The better bet is to treat a small soft 404 count that keeps rising as an early warning, and not to wait until the crawl numbers move.
What the team changed
The remediation plan put soft 404s first. Pages that did not exist began returning 404 or 410, and pages with real content were fixed so they rendered properly. The team then removed or noindexed the converter pages, tightened URL parameter handling, blocked low-value URL patterns in robots.txt, and added canonical tags for variant pages. Core Web Vitals work came last.
What to do
- Open the Pages report in Search Console and check the Soft 404 row over time. A count that keeps rising deserves a look even when the total is small.
- Sample the flagged URLs. If a page should not exist, make it return a real 404 or 410. If it should exist, find out why Google sees it as empty and fix the rendering.
- Confirm what the server actually returns for a removed URL:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/removed-page/
A result of 200 for a page that no longer exists is how a soft 404 starts.
- Do not redirect missing pages to the homepage. Do not block URLs that return 404 in robots.txt either. Google’s help page warns against both because they make the site’s structure harder for Google to recognize.
- After a domain move, check whether Google is still crawling the old domain. In this case, Google kept crawling the old domain long after the move because the redirects were incomplete.