Crawled - currently not indexed is rarely a crawl budget problem

An SEO who reviews technical audits argues on r/TechSEO that crawl budget is almost never the cause of “Crawled - currently not indexed” on a site under a few hundred thousand URLs, and ranks logs, sitemaps and status codes last in the debug order. The thread, posted October 31, leads with two cheaper checks: fetch the URL the way a bot would, as a plain request with no JavaScript, then diff that HTML against two or three pages in the same template that are indexed.

The second check is the one worth running first. Pages built from the same template share the same configuration, so a difference between an indexed sibling and an excluded one is unlikely to be technical, and the thread’s author reports it is usually content: near-duplicate boilerplate, a thin variant, the same intro paragraph across forty pages. That diff needs two browser tabs and no tooling, and it tests the expensive theory for free, because a page that differs from its indexed siblings only in content gives crawl-budget work nothing to fix.

The no-JavaScript fetch has a weaker case, because the reason the post gives for it did not survive the thread. The post claims a page whose main content appears only after hydration is sitting in the render queue behind everything else on the site. One commenter disputed both halves, saying pages get demoted from indexed to crawled-not-indexed with no JavaScript involved, and that Googlebot runs the JavaScript needed to render page content. The same reply names the better instrument: URL Inspection’s live test and its “view crawled page” view show the HTML Google actually got, which a plain request does not. Google itself has dropped its advice to test pages with JavaScript turned off.

A second commenter ranked status codes first rather than last, then gave the reason they belong last. A URL returning a 4xx, 5xx or 3xx never reaches “Crawled - currently not indexed” at all, that commenter wrote, because Search Console files it under not found or redirect instead. If that holds, the status is already evidence Googlebot got a 200, and logs and status codes confirm a diagnosis rather than produce one.

Click depth is the step in the order with no agreement behind it. The post treats five clicks deep, or a page linked only from a paginated archive, as a reason Google might crawl a URL and still not care about it. One commenter called depth irrelevant and said a Google search turns up pages seven levels deep in top positions, while another reported clearing the status with smarter internal linking and a maximum depth of three clicks, alongside authority building. The likelier picture is that a page reachable only from page nine of an archive carries a content problem the diff would surface anyway, so depth is best checked second.

The thread’s blunter framing is that this status often means Google crawled the page, understood it, and decided it was not worth storing, which makes it a content and internal-linking problem rather than a technical one, and no amount of sitemap pinging fixes it. That is a claim about Google’s intent rather than documented behaviour. The diff is still the cheaper bet, because it costs nothing and a crawl-budget engagement does not.

What the thread leaves open is how long to wait before treating the status as a verdict rather than a queue. Its author reports seeing pages flip to indexed weeks later with nothing changed on them, and no commenter named a threshold. Search Console’s reporting lag argues for patience as well, since indexing changes show up in performance data before the page indexing report catches up.

What to do

Run the cheap tests in this order before anyone scopes crawl-budget work:

• Compare an excluded URL’s HTML against two or three indexed URLs in the same template, and treat any content difference as the lead.
• Use URL Inspection’s live test and “view crawled page” to see the HTML Google got, rather than a browser view or a plain no-JavaScript fetch.
• Leave log files and status codes until the diff comes back clean.
• Give a recently affected page weeks before rewriting it, since the thread reports pages flipping to indexed on their own.