Daily index checks caught a 77K-URL redirect the day it shipped

The development team on a Magento migration bulk-redirected 77,493 legacy URLs matching a /catalogviewid/ pattern, all of them indexed at the time. A case study from Botpresso, a technical SEO agency, records the entire set flipping from “Indexed” to “Page with redirect” on September 3, the day the redirect shipped. The agency had loaded the client’s URL list into IndexVigil, its own index-monitoring tool, before the migration began.

Those URLs were earning. In the quarter before the redirect, the pattern pulled 8,230 clicks at an average position of 6.9. Magento had been serving the legacy pattern alongside the newer search-engine-friendly URLs and Google had indexed both, so the redirect did not clear out dead pages. It took out pages ranking on the first page of results.

Botpresso’s piece says Search Console confirmed the traffic decline but could not name which URLs had left the index, date the change per URL, or give a consistent reason for each removal. That is the Page indexing report doing what it was built to do, which is count pages by current status and show examples rather than keep a dated history of each URL. Once a drop surfaces in the Performance report, the per-URL evidence of what changed has already been replaced by the current state.

The case study promotes IndexVigil, and the tool is the least portable part of it. What made the loss datable was the list: 77,493 URLs enumerated before the change, each carrying a status recorded every day. Any method that produces that record can answer which URLs left and when, and nothing assembled afterwards can, because the comparison needs a before.

Diagnosis without those dates turns into argument from crawl exports and recollection, which is how a post-migration drop ends up attributed to theories about site quality instead of the release that caused it.

What to do

Four things make a drop like this datable rather than arguable:

• Enumerate the indexed URL set before the migration branch ships, from Performance exports, a crawl, or server logs, and keep it as a file with a date on it.
• Record an index status for each URL daily through the cutover window, so any change has a day attached to it instead of a week it might have fallen in.
• Diff the statuses the day after each release. A whole pattern changing at once is a template or rewrite rule, not a decision Google made, and a rule can be reverted the same day.
• Check a pattern-wide redirect against click data by URL before it ships. A legacy pattern can be indexed and ranking at the same time as its replacement, which is exactly the case here.

If none of this is in place yet, the pre-change export is the part worth doing first. A daily index check is worth little without a list that pre-dates the change it is meant to catch.