If a brand’s true rate of appearing in an AI answer is around 50%, you need about 97 runs of the same prompt to measure it to within ±10 percentage points. To show that a change moved it from 50% to 60%, you need about 388 runs of each version.
Most claims about AI visibility rest on far fewer.
Why one run tells you almost nothing
AI answers are sampled, not looked up. Ask the same question twice and you can get two different lists. The largest public test of this, by SparkToro and Gumshoe (27 January 2026), ran 12 prompts 60 to 100 times each across ChatGPT, Claude and Google’s AI, 2,961 responses in all. ChatGPT and Google’s AI gave the same list of brands in under 1 in 100 pairs of answers. What held steady was how often each brand turned up across many runs. Their conclusion was that visibility percentage across many runs “is a reasonable metric”, and ranking position isn’t.
So the useful number is a rate. A rate from a sample has a margin of error, and the margin depends on how many runs you have.
The maths, in one line
Each run either names the brand or doesn’t. That makes the appearance rate a binomial proportion. For a 95% interval of ±m around a true rate p, you need about
n = 1.96² × p × (1 − p) ÷ m²
runs. The worst case is p = 50%, because that’s where answers vary most. When you report a result, use the Wilson interval rather than the plus-or-minus shortcut, because it behaves properly near 0% and 100% and on small samples. The calculator does both.
How many runs, at a glance
95% confidence. Runs needed for one prompt on one engine:
| True rate | ±5 points | ±10 points | ±15 points | ±20 points |
|---|---|---|---|---|
| 10% or 90% | 139 | 35 | 16 | 9 |
| 20% or 80% | 246 | 62 | 28 | 16 |
| 30% or 70% | 323 | 81 | 36 | 21 |
| 50% | 385 | 97 | 43 | 25 |
Both tables as a CSV.
SparkToro’s 60 to 100 runs buys you about ±13 to ±10 points at a 50% rate. That’s honest for “this brand shows up about half the time”. It’s not enough to tell 45% from 55%.
Did the change work? That costs more
Comparing two rates (before and after a page change, or two versions of a page) needs more runs, because both numbers carry noise. Runs per version, to detect the change 80% of the time at the usual 5% significance level:
| From | To | Runs per version |
|---|---|---|
| 50% | 60% | 388 |
| 50% | 70% | 93 |
| 30% | 50% | 93 |
| 20% | 40% | 82 |
| 10% | 30% | 62 |
A ten-point lift needs hundreds of runs of each version. A twenty-point lift needs about a hundred. Below that, “it went up after we changed the page” is mostly noise.
Worked examples
A small sample
My P37 call missed because none of eight AI answers was stale 24 hours after the Bank Rate decision. The 95% Wilson interval for 0 of 8 runs from 0% to about 32%. So the data rules out a high stale rate. It doesn’t show the engines are never stale. With two runs per engine, one stale answer out of two (as Gemini gave at one hour) is consistent with a true rate anywhere from 9% to 91%.
A large sample
SparkToro reports a hospital appearing in 69 of 71 ChatGPT answers to one prompt. The Wilson interval is about 90% to 99%. That is a number you can act on.
The common case
A brand that appears in 3 of 10 runs. The interval runs from about 11% to 60%. “We’re in 30% of answers” is the midpoint of a very wide range.
The calculator
Enter the rate you expect and the margin you can live with; or what you got, as k of n, to see the real range; or two rates, to see the runs each version needs.
Where this understates the problem
The formulas assume each run is independent. Real runs often aren’t: same account, same hour, same data centre, possibly cached. Correlated runs carry less information than their count suggests, so the true margin is wider than the table says. Spreading runs across days and fresh sessions helps.
They also cover one prompt. People ask the same thing in many ways, and SparkToro also ran a separate set of 142 human-written versions of one headphones question for that reason. Variation between prompts is a separate source of noise, and more runs of one prompt don’t reduce it.
The counterargument
Tracker vendors would say nobody reads one prompt in isolation. They average across dozens or hundreds of prompts, and an average over many prompts is much steadier than any single one. That’s right for a headline figure like “share of voice across 200 prompts”. It doesn’t rescue per-prompt claims (“we now appear for ‘best accounting software’”), and it doesn’t help a buyer check the number unless the vendor says how many runs sit behind each prompt. None of the 13 trackers I checked publishes accuracy evidence.
What to ask of any AI visibility number
The same as the five labels: date, prompt, model, runs, one change at a time. This post puts a number on the fourth. If a figure doesn’t say how many runs sit behind it, you can’t work out its margin, and neither can whoever is quoting it.
Figures computed 6 November 2026. Normal-approximation sample sizes; Wilson score intervals for reported results; two-proportion sample sizes use the pooled normal approximation, two-sided α = 0.05, power 0.80.