Testers preferred AI-written search pages to Google's top result
Summary
A Google Research paper had a model write entire search result pages, and testers preferred them to Google's top-ranked result on 73.5 percent of information-seeking prompts, which leaves a rank tracker with no fixed list to measure.
Sample each tracked query several times a day and report visibility as a rate with its spread rather than a single position.
A Google Research paper had a model write entire search result pages, HTML, CSS and JavaScript included, and human testers preferred those pages to the top-ranked result on 73.5 percent of information-seeking prompts. The figure comes from iPullRank’s write-up of the paper, published October 15 by founder and CEO Mike King, who also rebuilt the paper’s architecture as a working tool.
The page gets assembled per query by instruction rather than designed once. “Generative UI: LLMs are Effective UI Generators” describes three parts:
- a tool server that gives the model Google Search and image retrieval
- a system prompt, roughly 2,450 words by King’s count, telling it how to read the query, plan interactive features and cite what it uses
- post-processors that clean the generated code before the browser receives it
What that system prompt tells the model to do matters more to visibility than the layout does. It requires searches for facts and entities before the page is written, citations for whatever gets used, and removal of placeholders when the data is not there. The likelier explanation for a brand appearing on one run and vanishing on the next, with nothing changed on its site, is that a citation on a generated page is an instruction being carried out rather than a ranking being displayed.
King’s proof of concept, Genraggle, reconstructs that architecture with Gemini and Google Search, and it is where the instability becomes visible. Running the same query repeatedly produced substantially different interfaces, with calculators, comparisons or simulators appearing depending on the run. Genraggle also perturbs the retrieved result set to see what a poisoned list does to the finished page, and it logs which sources end up cited.
King’s conclusion is blunt: rank tracking as the industry has historically done it cannot be trusted on its own, because “the page we are trying to measure is no longer a stable object.” That follows from the architecture rather than from the preference number. A position implies a fixed list in a fixed order, and a page written fresh for each query does not have one.
Neither the paper nor Genraggle is a live Google surface. Treating a Google research paper as a product roadmap has misfired before, as it did with DeepMind’s ranking paper that tested only reranking. The measurement advice survives that caveat because the instability is already in the numbers practitioners report on: Google has been serving different results to rank trackers on a measurable share of queries, which breaks a single daily capture the same way a generated page would.
What to do
King’s recommendations amount to measuring a moving surface by sampling it rather than checking it.
- Take several samples a day for each tracked query and judge each capture against the last verified one, which is how a swapped or poisoned result set gets caught.
- Report visibility as a rate with its spread across days and locations. In King’s phrasing, a figure with a spread is more honest than a position with a date on it.
- Use average position as a trend line only, not as a number anyone acts on week to week.
- For AI surfaces, run each query many times and report the share of runs that retrieve or cite the brand.
- Segment by whatever changes the page, which for now means location and persona.
- Anchor every tracker to Search Console and referral traffic, since a tracker sees one slice of a personalized page.
The reporting change is worth making before anything ships, because the surfaces being tracked already move.