AI models name a single cause when the key evidence is missing
Summary
A model with the deciding observation in front of it almost always gets an SEO diagnosis right. Without that observation, it still names one cause about half the time, and the wrong call tends to sound like routine advice.
Ask for the observation behind a diagnosis, the causes still in play, and what data is missing. Send the question to a strong model rather than the cheapest one available.
Twenty-seven AI models named the right cause of a technical SEO problem 99% of the time when the prompt contained the single observation that settled it. Delete that one observation and change nothing else, and only 51.5% of answers correctly reported that the evidence no longer supported one diagnosis, per the WARRANT-SEO benchmark published Thursday by the SEO consultancy iPullRank.
Each scenario exists in three versions. IPullRank built 27 diagnostic cases from what its write-up calls real-world diagnostic experience, then wrote one version carrying the decisive fact, one deleting it, and one that keeps the fact but adds “you can’t tell from this” to the options, to see whether a model abandons a sound call when handed an exit. The set comes to 81 questions and 6,427 scored responses. Patrick Schofield, iPullRank’s associate director of relevance engineering and the write-up’s author, says a standard accuracy score cannot catch the failure because it “just sees an answer and checks whether it matches.” The scored scenarios stay confidential so models cannot memorize them, which also means nobody outside iPullRank can rerun the set.
Take a made-up version of that setup: a site loses traffic the same week it ships a new product template, and the prompt mentions that only the pages on the new template declined. That detail rules out every competing explanation, and models handle it. Cut the line about which pages declined and the prompt now supports several causes equally well, yet close to half the answers still pick one.
The scores do not spread evenly across the field.
| Model | WARRANT score |
|---|---|
| GPT-5.5, Grok 4.5, Grok 4.6, Gemini 3.7 Flash, Claude Opus 5, Kimi K3, Muse Spark 1.2 | 1.000 |
| GLM-5.2 | 0.887 |
| Claude Sonnet 5 | 0.728 |
| Claude Haiku 4.5 | 0.556 |
| Mistral Large | 0.500 |
The write-up does not draw this line, but the spread lands hardest on automated work, because the model wired into a scheduled job or a bulk editing tool is usually the cheap one. Yoast already lets agents rewrite canonicals and robots flags in bulk, so a diagnosis nobody checked can become a few thousand edits.
The dangerous output is not the obviously wrong one. Schofield recommends giving the most scrutiny to the recommendations that sound the most ordinary, which is the right place to look: an unsupported diagnosis arrives as standard advice, a canonical to fix or internal links to add, and nothing in the wording reads as a guess. The model does not mention that it never saw the log files or the deploy date.
What to do
- Split the question in two. Ask what could be causing the drop, then separately ask what is causing it, which is the division iPullRank recommends.
- Ask for the evidence trail instead of the fix. Schofield’s suggested structure is a diagnosis, the observation supporting it, the competing causes still viable, and what is still missing.
- Treat “what is still missing” as the most useful line in the answer, because it names the data to pull next: the log sample, the deploy date.
- Put a second person on any call that sounds routine, particularly before it feeds a bulk change.
- Keep diagnosis on a model that scored well rather than the one an automation defaults to.