What do AI visibility trackers say about how they measure? 13 tools checked

Checked 30 October 2026

None of the 13 AI visibility trackers I checked publishes any evidence that its numbers are accurate, and none says whether it collects answers signed in or signed out.

I read each vendor’s own product, help and documentation pages and asked the same nine questions of every tool. This is a check of what each vendor tells you about its method. It is not a test of the data.

The headline numbers

• 13 tools checked: Semrush, Ahrefs, Profound, Peec AI, Otterly.AI, Scrunch, AthenaHQ, SE Ranking’s SE Visible, Similarweb, BrightEdge, Conductor, seoClarity and Rankscale.
• 0 publish validation or accuracy evidence for their own numbers.
• 0 say whether answers are collected from a signed-in or signed-out session.
• 3 say how many times a prompt runs on a given date. All three say once.
• 2 name the UK as a supported market (Semrush and SE Visible).
• 6 publish how their headline metric is calculated.
• 5 say on their own pages that the same prompt can give different answers.
• 2 (BrightEdge and seoClarity) answer none of the nine questions on their public pages.
• The best score is 5 of 9, reached by 6 tools.

The nine questions

1. Prompts: does it say where the prompts come from?
2. Collection: does it say whether answers come from the live app or an API?
3. Frequency: does it say how often prompts run?
4. Runs per date: does it say how many times each prompt runs on a given date?
5. Session: does it say whether collection is signed in or signed out?
6. UK: does it name the UK as a market?
7. Formula: does it publish how the headline metric is calculated?
8. Variability: does it say answers change from run to run?
9. Validation: does it publish evidence that its numbers are accurate?

Questions 4, 5 and 9 matter most, and they’re the ones nobody answers well. They’re adapted from Five labels every AI visibility number needs.

Tool by tool

Every row, with the page each answer came from and a note on what it says, is in the CSV.

Why runs per date matters

On 27 January 2026 SparkToro and Gumshoe published 2,961 runs of 12 prompts through ChatGPT, Claude and Google’s AI answers, collected by 600 volunteers in November and December 2025. They found less than a 1 in 100 chance of ChatGPT or Google giving the same list of brands twice. Some brands still showed up in almost every answer, so they concluded that a percentage across many runs is a usable measure and a ranking position isn’t.

So a “visibility” number is a sample. Its reliability depends on how many runs sit behind it. Three tools state a number: Profound and Rankscale say each prompt runs once per platform per update, and SE Visible’s example implies the same. The other ten don’t say. None says how many runs are needed before a week-on-week change means anything.

I’ve seen the same thing on a much smaller scale: the same prompt in a fresh chat gave a different shortlist.

Why app versus API matters

Two vendor studies, both published in 2026, compared answers from the live apps with answers from the same companies’ APIs:

Both come from companies that sell tracking, and Surfer notes it took one sample per prompt. The direction is still clear: an API answer is a different answer. Eight trackers say which they use. Most say the live interface; Ahrefs, Otterly.AI and Peec AI each mention at least one engine collected through an API, and Rankscale and Scrunch say they mix both.

Why signed in or out matters

Signed-in AI answers can be shaped by memory, chat history and account settings, which is why this site’s own studies state the session for every engine. A tracker collecting signed out is measuring an answer no signed-in customer sees, and the reverse. Semrush says AI answers are personalised enough that no platform can give exact visibility numbers, which is fair. Conductor says it tracks ChatGPT’s default setting. Nobody says which session they use.

Who says answers vary

Five say it in their own docs: Semrush, Peec AI, Otterly.AI, Scrunch and Conductor. Ahrefs calls its metrics directional rather than exact, which is honest, but says nothing about run-to-run change. Saying the data varies is the start. The next step is telling customers how much it varies for their prompts, and no metric definition I read includes a range.

What to ask a vendor before you buy

Copy these into the sales call:

1. How many times does each prompt run on each engine per date, and can I see every individual answer?
2. Are answers collected from the live app or an API, engine by engine?
3. Signed in or signed out? With memory and personalisation on or off?
4. Which country does each run come from, and can I choose the UK?
5. What's the exact formula for your headline score?
6. What's the run-to-run range on a prompt you track, and do you show it?
7. Have you checked your numbers against anything? Can I see the result?

A vendor who can’t answer 1, 3 and 7 is selling a number you can’t check.

Method

Every page listed in the CSV was opened on 30 October 2026, and only the vendor’s own product, help and documentation pages were used. Third-party reviews weren’t. A question counts as answered only if a vendor page says it in plain words; a sales claim with no detail (“ground-truth validation”, “identical to what your customers see”) doesn’t count as validation. Most pages were read through a tool that summarises them; Profound’s glossary, Peec AI’s documentation home and Semrush’s data page were then opened in a normal browser and matched. Each CSV row carries a note on every judgement, so you can disagree with a specific call. The rules every study here follows are on the method page.

Limits

• This checks documentation, not data quality. A tool can say nothing and still measure well, or say everything and measure badly.
• Paid-only help pages, sales decks and anything behind a login weren’t read. A vendor may answer a question privately to customers.
• The tool list is mine. It covers the best-known tools as of 30 October 2026, not the whole market.
• “Answered” is a reading judgement. The notes column explains each one.
• The SparkToro, Surfer and Genezio figures are those companies’ own; I haven’t rerun them.

What happens next

I’ll recheck the same 13 tools against the same nine questions on 30 April 2027 and publish what changed. If a vendor updates its documentation in the meantime and tells me, I’ll add a dated line here.

P35, called 30 October 2026: On 30 April 2027, fewer than 4 of these 13 tools will publish accuracy or validation evidence for their own numbers. Right if 0–3 do. Wrong if 4 or more do.

Download

CSV: 13 tools, nine questions, the page behind each answer. One row per tool. Free to cite with a link.

Related: Five labels every AI visibility number needs · Does ChatGPT give the same answer twice? · AI search statistics, traced to the source · Crawled is not cited

Sources

Every vendor page is listed in the CSV. Independent and comparison studies, each opened on 30 October 2026:

• SparkToro: AIs are highly inconsistent when recommending brands or products (27 January 2026)
• Surfer: Why you cannot track AI visibility through an API
• Genezio: Zero query overlaps between GPT-5.2 (API) and ChatGPT.com (4 February 2026)