Crawling & Indexing
Robots.txt, sitemaps, crawl budget, and indexing issues.
- News
Only 40% of domains persist across Common Crawl snapshots
Common Crawl modeled how much of the web its crawls capture and found a stable 40% core plus a churning shell, so absence from one crawl index proves little.
- News
A 200 while your site is down can turn pages into duplicates
John Mueller says downtime pages served with a 200 get their maintenance text indexed and deduplicated against other sites, and duplicates get crawled less.
- News
A CDN switch speeds up Googlebot while the cache is still empty
Google's Search Central blog explains why crawl capacity rises after a CDN cutover, and recommends pre-warming the cache, verified-bot allow rules, and 429s.
- News
Facebook's AI-written answer pages rank in Google and AI Mode
Meta has more than 6 million facebook.com/fb-answers/ URLs indexed, built from public Facebook posts, and Search Engine Roundtable reports no response from Google.
- News
Google says Crawled - currently not indexed may need no action
The rewritten Page Indexing report entry names crawler blocks as its one example, says nothing about quality, and sets a site-wide bar for filing an indexing bug.
- News
Google can keep a 4xx page in the index but drops its snippet
Google's Gary Illyes says Search held VLC's URL in the index through a 418 error, leaving only the link and a stale title, so snippet loss precedes ranking loss.
- News
Discovered - currently not indexed means Google never fetched it
A thread in r/TechSEO sets a triage order for the Page indexing report: date each URL, group by template, then split server problems from crawl demand.
- News
Crawled - currently not indexed is rarely a crawl budget problem
An r/TechSEO post puts a no-JS fetch and a template diff ahead of logs and sitemaps, and the thread's replies challenge the render-queue reasoning.
- News
Let Googlebot follow homepage redirects to show your favicon
Google's favicon doc, updated October 27, now says Googlebot must be able to follow redirects from the homepage to the page declaring the icon. Checks to run.
- News
Google says share links should stay out of Search by default
Google's Oct. 26 update to its user-generated spam guide tells platforms to keep share links out of Search by default, and only noindex does that reliably.
- News
Google Images and Discover previews do not support JPEG XL
Mueller said on Bluesky that Google's image SEO docs still omit JPEG XL, so serve .jxl as a picture source and keep a JPEG, WebP or AVIF file in the img src.
- News
One site's impressions hit zero while live URL tests still pass
A site owner says Search Console's Page Indexing report is stuck on October 12, with checks in server logs and Bing that separate stale data from deindexing.
- News
Google treats ?filter[]= and ?filter%5B%5D= as separate URLs
John Mueller, citing RFC 3986, says the raw and percent-encoded bracket forms are not equivalent, so faceted sites emitting both leave Google to dedupe them.
- News
Google's Indexing API docs no longer say how to raise quota
Google deleted the approval form and testing-quota wording from three Indexing API pages this week, but the 200-a-day publish cap is still documented.
- News
Google recrawls PDFs slowly, so publish updates at a new URL
Google's Martin Splitt and John Mueller explain how PDFs are converted to HTML, why embedded images miss Image Search, and why an updated PDF needs a new URL.
- News
Google now accepts UGC posts by API from approved platforms only
The UGC Fresh Data Program takes posts and engagement counts over OAuth 2.0, but requires stable per-post URLs, public attribution and on-page schema.org markup.
- News
Login walls in front of staging 301s hide them from Googlebot
Editorial backlinks landed on a staging subdomain with 40 indexed pages, and the fix is permanent page-to-page 301s Googlebot can fetch, plus outreach.
- News
Google documents how to tell Googlebot when to retry a 503
Google's reduce-crawl-rate guide now shows Retry-After examples in seconds and absolute UTC form, plus the 1-2 day limit on returning those error responses.
- News
A resolver without the new DNS root key breaks sites on Oct 11
The DNS root switches to a new signing key, KSK-2024, on October 11. Two sentinel queries defined by RFC 8509 show whether a resolver trusts it before the switch.
- News
Google puts numbers on site moves and core update recovery
The ranges reported from Gary Illyes' Barcelona session cover discovery, refresh, canonicalization and manual actions, with no sample size or definition of typical.
- News
A robots.txt rewrite to stop AI bots took a site out of Google
The rewrite allowed Googlebot but named Googlebot-Image and Google-InspectionTool, and Google's crawler docs show why robots.txt needs a per-variant audit.
- News
JavaScript error pages can make Google index another site instead
A Reddit user saw Google pick a casino page as canonical for their pages. Mueller says an indexed JavaScript error page may explain it and suggests hourly checks.
- News
Google says AdSense crawler rules also affect Ad Manager
Google's crawler docs now say Mediapartners-Google rules affect several ad products, and the crawler ignores the * group, so only a named robots.txt group stops it.
- News
Google crawled a news site less as its soft 404 count climbed
A Search Engine Land case study traces a 90% traffic loss at a crypto news site to a botched migration, then finds soft 404s growing on its other country sites.
- News
Google deindexed a 1,800-post blog that passed technical checks
A gaming blog's 1,015 pages sat at Crawled - currently not indexed with blank canonicals, and a Screaming Frog crawl found exploit scripts and download gate code.
- News
Fix soft 404s on empty category pages by how long stock is gone
Search Console flags empty category pages as soft 404s. Added content, a 302 or noindex each fits a different stockout length, and a canonical fits none.
- News
Search Console flags failed page resources that logs show loaded
A WooCommerce owner saw URL Inspection mark images and fonts as Other error while logs showed 200 OK. Replies point to the live test's limits and no-cache headers.
- News
Check server logs before blocking millions of news tag pages
An r/TechSEO thread on a news site with millions of tag and author pages weighs noindex against robots.txt disallow, and puts a server log check before both.
- News
OpenAI's search crawler now outpaces its training bot in logs
Botify's analysis of about 7 billion log events finds OAI-SearchBot up 3.5x since GPT-5, and OpenAI's docs show GPTBot blocks do not cover ChatGPT search.
- News
ChatGPT reportedly gets Google results from a scraping service
Botify says OpenAI, Meta and Perplexity reportedly buy scraped Google results from SerpAPI, and explains the jobs of GPTBot, OAI-SearchBot and ChatGPT-User.
- News
A noindex tag does not stop AI bots from using your bandwidth
Botify ties rising AI crawler traffic to bandwidth and uptime costs. Google's docs show why noindex cannot cut that load: bots must fetch the page to read it.
- News
Mueller can't confirm splitting sitemaps by age changes crawling
Mueller gave five reasons sites split XML sitemaps. File limits and hreflang size justify a split; splitting by content age is a theory he could not confirm.
- News
Blocking CSS and JS in robots.txt breaks how Google renders pages
An r/TechSEO poster saw 79% of crawl requests go to CSS and JS files. The replies say keep them crawlable, as Google's docs do, and find out why they are refetched.
- News
A wildcard DNS record got made-up subdomains indexed and ranking
A dating network's wildcard DNS left gibberish subdomains indexed and drawing traffic. Google's canonical docs back a path-preserving 301 to the www host.
- News
Indexing API got ordinary pages indexed fast in a Reddit test
A site owner's split test got 94% of Indexing API pages indexed in a week, against 8.4% by sitemap. Google supports the API only for job and livestream pages.
- News
Mueller: odd canonical picks often trace to what Googlebot saw
John Mueller lists why Google treats URLs as duplicates, from bot challenge pages and unrendered JavaScript shells to mobile comparison and parameter guessing.