Perplexity's crawler hit 70 English URLs and none in Turkish
Summary
On one site, PerplexityBot discovered 70 new English URLs in a single burst and never requested the Turkish versions listed as hreflang alternates in the same sitemap.
Split your AI crawler logs by language prefix and check whether the translated URLs are being requested at all.
PerplexityBot requested 70 URLs it had never touched before within 26 seconds on one site, according to an IP-verified log sample posted to r/TechSEO. Every one of them was an English URL. None of the site’s Turkish equivalents, listed in the same sitemap as hreflang alternates, were requested at all.
The site is a SaaS with a blog, around 84 English URLs in the sitemap and the same again under /tr/. Its owner, posting as u/woodoo139 on 1 November, logs every AI crawler request and checks the client IP against the operators’ published ranges, so these rows are verified PerplexityBot and OAI-SearchBot traffic rather than anything that merely sent the user-agent string. The burst came on 28 October. In the two months before it, PerplexityBot had taken 23 distinct paths in total. The new requests arrived from 8 IPs inside a two-minute window and ran across the whole site: blog posts, data and report pages, tool pages, and routes like login, signup and terms that no answer engine would cite. A slower round the next day added 31 more new paths, all of them og:image routes.
The Turkish zero is not a logging gap. Woodoo139’s logging runs in Next.js middleware on every page request before anything is served, and the /tr/ pages are ordinary page routes through it, so a request to one of them would have been recorded exactly like an English one.
The language split matters more to other sites than the burst does. If PerplexityBot only queues URLs it meets through links and never reads the hreflang pairs in a sitemap, then on a multilingual site the English page is the candidate answer source whatever language the question is in, as one reply in the thread argued. Google treats alternates loosely enough that it will serve an hreflang URL it never indexed on its own, and there is no reason to assume a retrieval crawler does even that much. The reply’s suggested check, which URL Perplexity cites for a Turkish question about the product, cannot be run yet. None of the 120 answers to the site’s four tracked Turkish questions cited it on any engine, so there is no cited URL to compare against the English one.
The trigger is unexplained, and woodoo139 says so. A robots.txt edit allowing two API paths went out about 10.5 hours earlier, and neither of those paths is in the batch. Guides had gone into llms.txt two days before that. OAI-SearchBot did a smaller version on 8 October, 20 new paths the day after a sitemap lastmod fix. Against all that, an x-default fix in late October was followed by nothing bigger than four new paths over the next three days, and the site edits robots.txt, llms.txt or its sitemap most weeks, so almost any burst will have a change sitting a few days behind it. Treating the correlation as a recrawl trigger is premature on a single site. A reply also pointed out that Perplexity documents a separate agent, Perplexity-User, for pages fetched because a person asked about them, so a PerplexityBot burst is not evidence that anyone was asking about this site.
What the logs cannot say is whether any of those requests got content. Each row keeps crawler, path, IP, verification result and time, and nothing about the response, so a 200 and a 404 cannot be told apart afterwards. Woodoo139 now claims only that the paths were requested, after calling them fetches in the thread’s title. A 200 in the log can still mean the crawler got an empty page, so response status and bytes sent belong next to the request.
What to do
- Split your AI crawler log by language prefix and check whether the translated paths are requested at all. If they are not, internal linking to those pages is the thing to test, because the sitemap’s hreflang pairing cannot be relied on to get them queued.
- Record response status and bytes sent per crawler request. Without them, a burst tells you what was asked for and nothing about what was served.
- Verify crawler hits against published IP ranges before reading anything into them, and keep the user-requested agents such as Perplexity-User and ChatGPT-User in a separate bucket from the index crawlers. The two answer different questions, one about demand and one about coverage.
- If your request logging sits in middleware that skips file-like paths, allow robots.txt, llms.txt and sitemap.xml through it explicitly. Otherwise a crawler’s re-read of robots.txt before a burst goes unrecorded, and the correlation cannot be checked.