AI visibility swings of 20 points can be noise at 10 runs a prompt
Summary
AI visibility percentages are estimates drawn from answers that change on every run, so two vendors' numbers mostly reflect their different methods. The useful reading is one fixed setup compared against itself over time.
Before reporting a month-over-month gain, find out how many times the tool runs each prompt. At low run counts, a single reading carries a margin of error wide enough to swallow a 20-point climb.
Two AI visibility platforms reported 41% and 23% presence for the same client. iPullRank’s September 17 piece on accuracy versus precision opens with that gap, and its author, Mike King, who founded the agency iPullRank, argues that neither number is wrong. There is no true value for either tool to miss.
King borrows the distinction from measurement science: “Accuracy is how close a measurement is to the true value. Precision is how close repeated measurements are to each other.” AI answers are generated probabilistically and personalized per user, so no single correct response exists for a tracker to match. What a tool can deliver is precision, meaning the same setup gives roughly the same reading when it runs again. A comparison between two vendors therefore says more about their methods than about the brand.
One prompt, many answers
ChatGPT’s answers to the same question change a lot from run to run. King cites research by Search Engine Land contributor Mike Sonders, who ran 12 buyer-intent prompts 100 times each through logged-out ChatGPT. About 44 brands appeared across the 100 runs of one prompt, while a single response named about 10. Only about five brands per prompt appeared in 80% or more of the responses. King’s summary is that showing up for a given prompt on a given day “is closer to a dice roll than a ranking.”
Because each answer is a random draw, a visibility percentage is a sample estimate, and its reliability depends on how many runs sit behind it. King publishes the margin of error for a single prompt whose true appearance rate is 50%:
| Runs per prompt | Margin of error |
|---|---|
| 5 | ±44 points |
| 10 | ±31 points |
| 20 | ±22 points |
| 50 | ±14 points |
| 100 | ±10 points |
Take a made-up example. A tracker runs a prompt 10 times a month. A brand appears in 5 of those runs in August and 7 in September. The dashboard shows a climb from 50% to 70%. At that run count the margin is ±31 points, so the 20-point climb sits inside the noise. A report crediting it to a content change is crediting chance.
Why two vendors land far apart
Sampling noise explains only part of a gap like 41% against 23%. Vendors also make different methodology choices, and each one changes what gets measured:
- Model mode. A Semrush study run with Kevin Indig of the Growth Memo newsletter sent 100 prompts through ChatGPT at minimal and high reasoning, which match Instant and Thinking in the interface. Only 25.6% of cited domains overlapped between the two modes.
- Prompt phrasing. A working paper on SSRN by Ehrlinspiel, Landwehr and Rudzki analyzed about 38,000 responses across five engines. Ranking-style prompts surfaced up to 20% more brands than open-ended ones, so a prompt library heavy on “best X” questions reports higher presence for the same brand.
- Collection method. Collecting through the API measures controlled model behavior. Scraping the consumer interface measures what users see, including product features the API never runs. Google AI Overviews has no developer API, so a vendor claiming API collection there is approximating or scraping.
- What counts as a mention. A brand name in body text, a linked citation and a spot in a ranked list are different events. Tools that count them differently get different percentages from identical responses.
Account tier is one more input a vendor may or may not hold fixed, since ChatGPT’s free and paid tiers cite different sources.
The practical conclusion is that one fixed setup, compared against itself over time, is the only reading with a known error.
What to do
- Freeze the setup before the next reporting cycle. Write down the model, mode, account state, geography, prompt set and runs per prompt. Any change to those inputs starts a new trend line.
- Report movement inside one tool. Stop quoting a second vendor’s number in the same deck, since the two are not measuring the same thing.
- Test precision yourself. Run the same prompt set twice under identical conditions. If the two runs disagree by more than the movement you plan to report, that movement is noise.
- Put King’s methodology questions to your vendor:
- Which model and mode are pinned, and how are version changes announced?
- How many runs per prompt does each collection use, and does the dashboard show a range rather than one number?
- Where does collection originate geographically, and can you control it?
- Is there a methodology changelog, and can you export raw responses rather than extracted metrics?
- Check visibility against referral data before acting on it. King calls visibility metrics “channel metrics, and they’re proxies. Performance metrics are the actuals.” If presence moves and sessions from AI assistants do not, the likelier explanation is noise.
Watch out for
Logged-in collection accounts drift. ChatGPT’s Memory feature lets a collection account learn from its own prompt history, so the instrument changes what it measures over time. Logged-out collection avoids the drift but misses the personalization most real users get.