Most AI-visibility checks are a single screenshot: we asked ChatGPT and we were in the answer. Before publishing any engine comparisons ourselves, we wanted to know how much that single ask is worth — so we took our plumber vertical and re-ran the identical flagship queries multiple times per engine (four samples each, including the original run).
- The lists genuinely differ run to run. Between repeats, Gemini's answers shared on average about two-thirds of their businesses (mean Jaccard overlap 0.657); OpenAI's shared about half (0.484).
- Point numbers are weather. Gemini's flagship named 25 businesses in our original run, then 17–20 on repeats. Any single-run figure — including ours — deserves a range around it.
- Ordinal findings held. Gemini out-surfaced OpenAI in all four samples (17–25 businesses vs 11–15) — so "Gemini consistently reaches more" is safe to say, while "Gemini reaches twice as many" would be over-reading one sample.
- The stable core is small. Of all plumbers Gemini ever surfaced across the repeats, only 13 of 29 appeared in every run. For OpenAI, 5 of 22. Two businesses — Mr Plumber and Response Plumbing — sat in both engines' stable cores.
Surfacing once is not being on the shelf
This reframes what AI visibility even means. The shelf isn't a fixed leaderboard — it reshuffles on every ask, the way a market stall rearranges overnight. What persists is the core: businesses whose presence in the sources is strong enough that every assembly of the answer includes them. One appearance in one run tells you that you can surface. Appearing in the core tells you that you reliably do — and that's the difference between a lottery ticket and shelf placement.
(Honest caveats: this calibration is small — four samples per engine, one vertical, flagship tiers only — so treat the overlap numbers as indicative rather than precise. The repeats also fold in about a week of search-index drift, so some churn is the web moving, not just model randomness. Both effects argue for the same conclusion — never trust a single snapshot, including this one.)
Test yourself more than once before celebrating or panicking. One appearance — or one absence — is a coin flip's worth of evidence.
Aim for the core, not the cameo. The stable-core businesses were the ones multiple independent trusted sources agreed on. Depth of citation beats breadth of one-off mentions — which loops back to being written about.