AI Crawler Management
Blocking training bots, allowing retrieval bots, AI-specific robots.txt.
-
Managed WordPress hosts silently block AI crawlers
Managed WordPress hosts block AI crawlers by default, preventing your content from appearing in ChatGPT search and Perplexity results. Check robots.txt now.
-
Google's Web Bot Auth adds cryptographic bot identity
Google's Web Bot Auth adds cryptographic signing to HTTP requests, replacing spoofable user-agent headers with verified bot identity that works across IP ranges.
-
Cloudflare now enforces canonical tags as 301s for AI crawlers
Cloudflare converts canonical tags into 301 redirects for AI crawlers, forcing them to follow preferred URLs instead of treating canonicals as optional hints.
