AI visibility is real and worth measuring. AI ranking is not, because there is no ranking there to measure.
Understanding that difference is the difference between a monthly report that helps you and an expensive one that tells you nothing.
The invoice arrives first
Software that promises to track your brand’s visibility inside AI answers runs from around $99 a month to $449. Some of these tools are useful. Most are sold with a promise they cannot keep: that they will show you your position and track it over time.
Ask the same question twice, get a different answer
The largest test of this came from SparkToro, which had 600 volunteers run 12 questions through ChatGPT, Claude, and Google’s AI a combined 2,961 times.
There is less than a one in one hundred chance that asking the identical question twice returns the identical list of brands. For the same brands in the same order, it is closer to one in a thousand.
That dismantles the premise of rank tracking. If a dashboard says you moved from position four to position two, it has told you nothing. Running the query again would have produced a different number with nobody doing any work.
The distinction that will save you money
Measure whether you are in the room. Do not measure where you are standing.
The same study included an example worth studying. When people asked about cancer care on the West Coast, City of Hope appeared in 69 out of 71 answers, a 97% presence rate. But it was named as the top recommendation in only 25 of them.
Two different facts live inside that one result. The presence rate was high, stable, and genuinely meaningful. The ordering was random noise. A dashboard reporting average position would show a mediocre number bouncing around month to month, hiding the fact that this organization is winning.
How many times do you have to ask?
A team at the University of St. Gallen studied exactly this. Their paper is titled, helpfully, “Don’t Measure Once.”
A single observation, they found, is misleading. A usable read takes at least seven runs of each question per day, aggregated over two to four weeks. The reason for that window is striking on its own: the specific websites these engines pull from change by roughly 65% from one day to the next.
So ask any vendor how many times per day they run each question and over what window they aggregate. If the answer is once, you are buying a screenshot with a subscription attached.
The part nobody expects
Another researcher analyzed nearly 13,000 answers about 20 companies to find what was driving the variation.
The largest factor was not the company being asked about. It was how the question was worded, which accounted for more than a quarter of all the variation. Which company was being asked about accounted for 1.5%. The same work found that asking a question a sixth or seventh time added almost nothing, while varying the wording improved accuracy many times more.
The practical rule is therefore the opposite of what most tools do. Test many phrasings of your buyer’s question rather than many repetitions of one. Repeats are cheap to automate, which is why tools favor them. Phrasing coverage requires someone to think about how your customers actually talk.
So are the tools worthless?
No, and it is worth being fair. Digiday interviewed agency and brand leaders about these platforms, and one summary stood out, from Ryan Mason of Markacy: “There’s really not much an AI tool can do or tell you to do. It’s just a benchmarker.”
A benchmarker has real value. It gives a directional read on presence and tells you which outside websites are being cited in your category. What it cannot do is tell you what to fix. Buy it as an instrument, not a strategy. A thermometer is worth owning. It is not worth what a doctor costs.
A standard worth defending
These five rules work whether you buy software or do it by hand, and they give you something to hold a vendor to.
One. Measure presence, not position. Whether you appear is real. Where you appear is noise.
Two. Vary the wording more than you repeat it. Five well-chosen phrasings beat fifty repetitions of one.
Three. Aggregate over two to four weeks. Never decide from one day.
Four. Track which sources get cited. This is the only output that tells you where to go work. If the same three outside websites keep appearing in answers about your category, you have just been handed your public relations target list.
Five. Tie it to pipeline, or treat it as unproven. A score that never connects to an inquiry or a sale is a vanity metric wearing a lab coat.
Rule four turns this from an anxiety exercise into a work plan. Everything else tells you where you stand. That one tells you what to do.
We wrote in June that if you cannot tie a win to a real business outcome, treat it as unproven. Nothing about this technology being new changes what counts as evidence.
The risk is not that these tools are fraudulent. It is buying a precision instrument for a question with no precise answer, then making budget decisions from the readout.
Want a defensible read on how visible your brand is inside AI answers?
In a Brand Awareness Consultation for AI Engines, we run your category’s real buying questions across ChatGPT, Gemini, Perplexity, and Google’s AI search, across multiple phrasings, then report presence rather than position and show you which sources are shaping the answers.
No proprietary score. No dashboard subscription. A clear read and a work plan.
Sources
SparkToro. “New Research: AIs are highly inconsistent when recommending brands or products.” https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/
Schulte, J., Bleeker, M., and Kaufmann, P. “Don’t Measure Once: Measuring Visibility in AI Search (GEO).” University of St. Gallen, April 2026. https://arxiv.org/pdf/2604.07585
Žatuchin, D. “Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers.” arXiv. https://arxiv.org/html/2607.13304
Digiday. “Marketers question expensive AI visibility tools as inconsistent results fuel skepticism.” https://digiday.com/marketing/marketers-question-expensive-ai-visibility-tools-as-inconsistent-results-fuel-skepticism/