Screaming Frog 24.0 adds MCP server and uncrawlable link detection
Summary
Screaming Frog 24.0 ships an MCP server that lets AI assistants trigger crawls and export data via natural language, an auto-compare feature that diffs scheduled crawls and emails the delta, and uncrawlable link detection that flags non-standard markup like onclick handlers and href on divs. MCP feature parity is incomplete. Test on non-critical workflows before building pipelines around it.
What happened
Screaming Frog SEO Spider version 24.0 shipped on May 19, 2026 with six notable features. The headline addition is a Model Context Protocol (MCP) server that lets practitioners run crawls, export data, and build visualizations through AI chat assistants like Claude and LM Studio. The release also includes automatic crawl comparison for scheduled crawls, improved email notifications with issue deltas, email export attachments, uncrawlable link detection, and usage stats.
MCP is an open protocol from Anthropic that connects AI assistants to external tools and data sources. Screaming Frog’s implementation lets you issue natural language commands to trigger crawls, manipulate crawl data, and generate visualizations without touching the UI. The team notes that “not every feature and function is currently supported,” with plans to extend based on user feedback. This follows a broader trend of SEO tools adding AI assistant integration.
Why it matters
The MCP integration is the most significant change because it collapses multi-step workflows into single prompts. Consider a 50K-page e-commerce site running daily scheduled crawls. Previously, the SEO manager exported CSVs, ran scripts, and uploaded results to Slack. Now they can ask Claude to compare the last two crawls and surface product pages that lost H1 tags, getting a formatted summary without opening Screaming Frog directly.
For agencies managing multiple client sites, the combination of MCP, auto compare, and email notifications creates a monitoring pipeline that previously required custom scripting. You can schedule crawls, auto-compare them against the previous run, and have the delta emailed to clients with structural changes highlighted (new 404s, lost canonicals, redirect chains).
The uncrawlable links feature addresses a real blind spot. Screaming Frog now detects non-standard link patterns: <span href="...">, <div href="...">, <a onclick="goto('...')">, and <a href="javascript:...">. The naming is precise: “uncrawlable” here means these patterns should not be relied on for passing link signals, even though Google may still discover URLs from them. The release post notes that Google will attempt to parse anything in HTML that looks like a link, but warns against relying on non-standard patterns for link equity. The href attribute is not valid on <span> or <div> per the HTML spec, and onclick/javascript: URIs bypass standard link semantics. Google’s link best practices documentation specifies that standard <a href="..."> elements are what constitutes a proper crawlable link.
The auto crawl comparison feature eliminates the manual step of diffing two crawl exports. For scheduled and CLI crawls, Screaming Frog now compares the last two crawls in a project automatically and surfaces all changes. The delta also appears in email notifications, so you can spot issues like a sudden spike in non-indexable pages or disappearing page titles without opening the tool.
What to do
Try the MCP server on a non-critical workflow first. Feature parity is incomplete. Advanced crawl options like custom headers, JavaScript rendering settings, or proxy configurations may not be exposed yet. Start with simple tasks (summarizing a completed crawl, exporting a specific report) before building automation you depend on.
Keep crawl configs consistent across scheduled runs. The auto compare feature compares the last two crawls in a project. If you change crawl rules between runs (different include/exclude patterns, different spider settings), the delta report may flag false positives. Pages that were filtered out could appear as “removed” even though they still exist. Lock your config before enabling auto compare.
Enable “Store” for uncrawlable link detection. The release post states the feature requires “Store” to be enabled under Config > Spider > Crawl. Once enabled, check the Links tab and filter by “Pages With Uncrawlable Internal Outlinks.” The detection covers common patterns but won’t catch every edge case, such as links behind complex click event handlers or dynamically injected URLs. Treat the results as a starting point, not a complete audit.
Set up email notifications with the new delta reporting. If you already use crawl completion notifications, the upgrade adds issue tables and crawl comparison data to those emails automatically. You can also attach specific exports (like broken links reports) and send them to developers or stakeholders on a schedule. Attachments are zipped, and the system warns recipients if there is an error, such as files being too large for the email provider.
Understand the MCP’s local setup before wiring it into shared environments. The MCP server is bundled with Screaming Frog and launched from within the application. Before building shared automation around it, verify the listening interface and how authentication is handled in your environment.
Watch out for
High-churn sites will flood auto compare with noise. Sites with lots of short-lived pages (campaign landing pages, inventory that rotates daily) will generate massive deltas on every scheduled crawl. The release notes don’t appear to document whether URL patterns can be excluded from the comparison; you may need to filter the results manually or keep auto compare off for these projects.
MCP commands during active crawls. It’s unclear what happens if you issue an MCP command to start a crawl while Screaming Frog is already crawling. Test this before building any automated pipeline that could trigger overlapping crawls.