Log File Analysis
Crawl behavior, bot identification, and crawl budget insights.
- News
Cloudflare now filters logs with SQL before exporting them
Logpush Transformers run SQL on each batch before delivery, cutting export volume. Filter crawler logs on paths, not user agents, or unlabeled AI fetchers vanish.
- News
A new site's first AI visitor was a live fetcher, not a crawler
A five-day server log on a site launched October 7 shows GPTBot taking the whole sitemap in a single pass, nine ChatGPT-User fetches, and no AI referrals.
