AI visibility trackers built on model APIs miss what users see

A practitioner posting as u/Biffxs described building an in-house AI visibility tracker in a r/TechSEO thread, after deciding the paid tools cost too much for what they do. The first lesson listed was that the model APIs do not show what a user actually sees, because they return different sources and run no live search, so the tracker pulls its answers out of the consumer apps through a scraping API instead.

Those two surfaces answer different questions. A plain API call, with no search tool switched on, returns what the model’s weights hold, with no retrieval step and no cited pages attached, while the ChatGPT, Gemini and Copilot apps search the web on many prompts and list the pages they used. So a tracker built on raw API calls measures what a model remembers about a brand, which is not what the product tells someone who asks today.

The second lesson was that a single run means nothing. Biffxs reports the same prompt naming different brands every time, and runs each prompt five times per engine, then averages the most recent runs. Five is a floor rather than a target. China’s advertising industry standard sets 30 runs per prompt as its reporting minimum, and anything going in front of a client should sit nearer that end.

u/marintkael, replying in the thread, added that each run should also record whether the answer actually triggered a web search. An answer from the model’s own memory and an answer that searched and cited pages are measuring two different things, and ChatGPT does not search on every prompt, so the same question can flip between the two from one run to the next. Averaging across that mix produces a number that describes neither state. On ChatGPT the searched runs are at least identifiable, since its fanout requests name the result source and a freshness window.

Brand matching failed in both directions. Matching on domains missed multi-word brands and fired on brands named after common words, so Biffxs moved to a brand list with aliases and a human review queue for the unclear cases.

The part of the system worth building first is the citation list. Biffxs’s own gains came from reverse engineering which pages each engine cites for their prompts, then getting placements on those sites and listicles. U/lew1s666 reported the same ordering: the score for how often a brand gets named is noisy, the URLs that keep showing up across runs are what a team can act on, and that list works as an outreach queue. A mention score tells a client which way the line is moving. The recurring cited URLs tell the team which sites to pitch.

What to do

• Scrape answers from the app surface rather than calling the model API. Biffxs uses a scraping API, SearchApi, at cents per prompt. Another commenter runs the prompts locally through a browser simulation, which takes hours and costs nothing.
• Run each prompt at least five times per engine and average the recent runs. Report nothing off a single answer.
• Store a flag per run for whether the answer searched, and keep searched and unsearched runs separate rather than blending them.
• Match brands against a maintained alias list, with a review queue for ambiguous hits, instead of matching domains.
• Export the URLs that recur across runs and treat that file as the outreach target list.

Biffxs says the home-built numbers land close to what the paid tools report, which is the builder’s own unmeasured comparison. Anyone planning to drop a per-seat tool should run both against the same prompt set for a few weeks before cancelling.