How Accurate Is AI Visibility Data? Eight Checks Before You Trust the Numbers
AI visibility data is reliable when it rests on many fresh answers collected under stated conditions. Eight checks to run on any tool, export or agency spreadsheet before you trust a number.
Prefer Maya AI in Google
Highlight our stories in Search, AI Mode & AI Overviews.
AI visibility data is accurate enough to act on when it rests on many fresh answers collected under stated conditions, and misleading when it rests on one answer or on repeats of old ones. Before you buy a tool or report a number, ask how each answer was collected, how many were counted and whether you can open them. Maya publishes those conditions for its own data. The checks apply globally, in any market and language, except where a tool cannot show raw answers.
Below you will find how to tell whether a tool records real answers, why the same prompt gives different answers, where the answer behind a number comes from, eight checks to run on any dataset, and what to change when a number fails one.
Do AI visibility tools just guess what ChatGPT says?
A marketing manager who sees a mention rate for her brand on a dashboard usually has one doubt: did anyone actually ask ChatGPT, or is the number modeled? A tool that sends questions to AI assistants and records the answers is counting, and its mention rate comes from those recorded answers. Some products instead show a pre-collected index of answers to questions they chose, so it is worth asking whether your numbers come from your own questions. The eighth check below, opening the raw answers, shows which kind you have. The next question is whether the recorded answers are a fair sample.
One public test of that question comes from SparkToro. In a study published on January 27, 2026, about 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI answers a combined 2,961 times in November and December 2025 (SparkToro). For ChatGPT and Google's AI, the study put the chance that two answers list the same brands at under 1 in 100, with Claude only slightly more consistent, and the chance of the same order closer to 1 in 1,000. The authors started as skeptics of AI tracking and disclose that one researcher works at a company in that sector.
Their conclusion splits in two. Tracking a brand's position in an AI list was close to meaningless. Tracking how often a brand appears across many runs of many prompts was, in their words, a reasonable metric. The split between rate and rank is the basis of every check below: a visibility percentage can be trusted when it is built from enough answers, and a single answer or a rank cannot.
Why the same prompt gives different answers
AI answers are generated fresh each time, so two runs of one question can name different brands. Maya's guide to the seven AI visibility metrics explains why one run is a sample and how repeated runs produce a stable reading. The present article does not repeat that method. Variation from sampling is only one of several reasons answers differ, and the others are the ones a buyer can check.
Other factors that cause tracking results to vary
Google's own documentation is an example. Google states that AI Mode and AI Overviews may use different models and techniques, so the answers and links they show will vary, and that AI Overviews often do not appear at all (Google Search Central). A dataset that mixes the two Google surfaces, or counts a missing AI Overview as "brand not mentioned", reports a different thing from one that keeps them apart.
Reliability when AI engines change
Engines also change between reporting periods. Maya's research on ChatGPT answers documented changes in how sources were displayed and how answers were formatted within two months (Maya research). A tool can collect without errors through such a change and still produce a trend line that breaks. Reliable data marks the date of each engine change so that a drop after that date is read against a new baseline.
Where the answer behind the number comes from
Two tools can ask the same question and record different answers because they collect in different ways. Some send the question through an API, the programming interface developers use. Others collect from the consumer chat interface, the same screen a buyer uses. Some count a stored answer again instead of asking anew. Each route records a real answer, and each records a different kind of answer.
The table compares five routes on what the recorded answer can stand for.
| Collection route | What the answer reflects | Live web search | Personalization and location | Counts as a new observation |
|---|---|---|---|---|
| A manual check in a signed-in chat account | What one person was shown at one moment | Decided by the product for that session | Can be shaped by account memory and the user's own location | Yes, but a sample of one |
| An API call without the web search tool | The model's answer without live web retrieval | Off unless the web search tool is added to the request | Not stated in the documentation we read | Yes |
| An API call with the web search tool | The model's answer with optional live web sources | Optional with automatic tool choice; forced only when search is required | Approximate user location can be set | Yes |
| Consumer interface collection in a clean session | The platform's answer to the question as written, without user history | As the consumer product decides for that question | No signed-in account; country and language set per question | Yes, if collected that day |
| A cached or replayed answer | An earlier answer shown again | Whatever the original collection did | Same as the original collection | Not a new observation: the same collection event shown again |
The API rows rest on OpenAI's documentation: web search has to be enabled as a tool in the request, the model may choose not to search when the tool choice is automatic, and an approximate user location can be passed to refine results (OpenAI API docs). The clean-session and replay rows rest on Maya's methodology. The practical reading: ask which route produced each answer before you compare two tools' numbers.
Eight checks before you trust an AI visibility number
The checks below work on a vendor demo, an export from your current tool, or a spreadsheet an agency sends you. For each one, the question to ask is in bold, followed by the evidence to look for and what the number loses without it.
- How many fresh answers are behind this rate? Ask for the count of answers, not only the percentage. Without it you cannot tell a stable rate from a lucky week.
- Are replayed answers kept out of the count? A stored answer shown again adds a row but no new information. If replays are counted, the sample looks larger than it is and real changes look smaller.
- Which route produced each answer? Each record should say whether it came from the interface or an API. A tool that cannot say this cannot explain why its numbers differ from what a buyer sees.
- Was live web search on? For API collection, ask whether the web search tool was enabled and whether search was forced. Answers without search cannot cite your current pages.
- Which country and language were used? Ask whether both are fixed per question and kept the same across periods. A change in either changes the population of answers.
- Was any answer collected in a signed-in account? Account history can shape what one person sees, as the guide to AI visibility reporting for CMOs explains. A clean session keeps the answer about the question alone.
- Does every record carry a date and platform? Dates let you separate an engine change from a change in your own visibility. A total without dates cannot be compared across periods.
- Can you open the raw answer behind any mention? Pick five to ten records at random and read the answer text. You will see whether the brand is really named, whether a similar name was matched by mistake, and whether failed or empty answers were counted as "not mentioned".
How many answers a reliable rate needs
A rate always comes with a range of plausible true values, and the range narrows only as answers accumulate. Take an observed mention rate of 30%. Using the Wilson score interval at 95% confidence (our calculation, using the formula Edwin Wilson published in 1927), 3 mentions in 10 answers is consistent with a true rate anywhere from about 11% to 60%. At 9 in 30 the range is about 17% to 48%. At 30 in 100 it is about 22% to 40%, and at 90 in 300 about 25% to 35%.
So a move from 30% to 40% means little on 30 answers and something on 300. Maya's article on calculating AI visibility metrics shows the same effect from the other side, as the swing one extra mention causes in a small sample. In practice, read a rate together with its answer count, and compare periods only when both counts are large enough for the ranges to separate.
How Maya documents these checks
If you evaluate Maya, you can check its answers to the eight questions on its methodology and trust pages. According to those pages, Maya sends each question exactly as written, collects each tracked question once a day on each platform, and collects ChatGPT through its consumer interface in an anonymous session with no signed-in account. Country and language are set per question.
Maya stores whether each answer came from the platform interface or its API, and leaves failed and empty results out of the denominator instead of counting them as "not mentioned". Banking and investment projects are collected without a cache. On other projects an answer may be reused for a short window, three days by default, and every reused answer is flagged. The full answer text and source links are stored with the question, platform and date, and a mention rate can be opened down to the answers behind it. Maya's methodology also states that transparent measurement does not mean 100% accuracy: it means the basis and limits of a number can be examined.
Situations where the number is not ready to report
Most doubts about AI visibility data come up in a few recurring situations. The table pairs each one with what to check first and what to do before the number goes into a report.
| Situation | Check first | What to do |
|---|---|---|
| A colleague runs the same prompt in ChatGPT and gets different answers from the dashboard | Whether the colleague was signed in, and how many answers the dashboard rate rests on | Treat the colleague's answer as one sample and compare it with the dashboard trend |
| A vendor says its tool tracks how often Claude or Gemini mentions your product, but cannot say whether answers come from the API or the interface | The collection route field on a sample of records | Ask for records with the route marked before you compare the tool's accuracy |
| Your mention rate jumps after a quiet month with few new answers | Whether replayed answers were counted, and the answer count in both periods | Recount on fresh answers only and add the range |
| Rank tracking shows your brand moving from third to fifth in AI answers | Whether the position comes from one answer or an average across many | Report how often the brand appears instead of its position |
| The data looks inconsistent after an AI engine changed how it shows sources | The date of the engine change against your trend line | Mark the date and start a new baseline from it |
| You want to validate AI visibility data against real customer traffic | Whether the tool's question set matches the questions buyers ask about your category | Compare trends over the same periods and state that the two measure different things |
The rows that point to replays, routes and answer counts can usually be fixed inside the data. The other rows change how the number is reported.
What to change when a number fails a check
A failed check rarely means the data is worthless. A failed check means one part of the collection or the report needs to change before the number supports a decision. The changes below are tests: make one, record the date, and read the same question set again in the next period.
Recount the sample without replays
If replayed answers are in the count, rebuild the rate from fresh answers only and report both counts. A rate that stays similar after the recount is more trustworthy than before. A rate that changes a lot was inflated by repeats, and earlier reports based on it need a note.
Put the answer count and the range next to every rate
Add the number of answers and the plausible range to each rate in your report. Leadership can then see at a glance which moves are inside normal variation. A move that stays inside the range should not trigger a content decision on its own.
Fix the collection conditions before you compare periods
If the route, web search setting, country or language changed during the period, compare only the answers collected under the same conditions. Record each change with its date. The enterprise AI visibility platform buying guide covers the same problem for buyers who mix a pre-collected index with their own questions.
Ask for raw answers before you sign
In a demo, ask the vendor to open the answer text behind five mentions you choose. If the vendor cannot open them, you cannot check a single number in the dashboard. Maya's article on choosing an AI visibility tool lists five test questions to use in the same demo.
Illustrative example: a mention rate in a vendor demo
The following example is illustrative and does not describe a real customer or vendor. A marketing manager sees a demo dashboard showing a high mention rate for her brand on comparison questions.
Observation. The dashboard shows a rate but no answer count, and the export has several identical answers on consecutive days for the same question and platform.
Possible cause. One explanation is that replayed answers are counted as new ones, which would make a small sample look large. Another is that the questions ran through an API without web search, so the answers reflect the model's stored knowledge rather than current pages.
What Maya shows. In Maya, each record carries its platform, date and collection route, and reused answers are flagged, so fresh and reused answers, and interface and API answers, can be told apart.
Action and check. The manager recounts the rate on fresh answers only and asks the vendor which route and search setting produced each answer. If the recounted rate holds with a narrow range, she reports it with its answer count. If not, she reports the range and waits for the next period.
Limits of this checklist
The checklist does not compare named AI visibility tools and does not score any vendor's accuracy. Maya's article on choosing an AI visibility tool covers the buying decision. The checklist does not tell you how many questions your set should contain. The guide to building a prompt set does. Nor does the checklist connect visibility to revenue. The intervals shown are for a single rate on independent answers and are a simplification when answers from many questions are pooled.
Frequently asked questions
How can I validate AI visibility data against real customer traffic?
Compare trends rather than totals. AI visibility counts appearances in sampled answers, while analytics counts visits, so the two will not match one to one. If both rise or fall over the same periods on the same topics, that is consistent with the visibility data reflecting real exposure, but it does not show that one caused the other.
What data sources do AI visibility tracking tools use?
The source is the AI answer itself, collected either through a platform's API or through its consumer interface, plus the links shown in that answer. Some tools also offer a pre-collected index of answers to questions they chose. Ask which of these each number comes from.
How often should AI visibility data be collected and analyzed?
Daily collection gives enough answers to read a trend. Analysis is more useful weekly for the team and monthly for leadership. Reading every daily change invites reactions to normal variation.
Conclusion: Trust the conditions behind the number, not the number alone
AI visibility data from recorded answers is a sample of what AI assistants say. A number deserves trust when you know how many fresh answers it rests on, which route and settings produced them, and when you can open the answers yourself. A number that fails a check can usually be repaired by recounting, adding a range or fixing the collection conditions.
Which of the eight checks can your current data pass today? Maya publishes how it collects, counts and stores every answer on its methodology and trust pages, so you can run the same checks on Maya before you rely on its numbers.
Sources
- SparkToro, New Research: AIs are highly inconsistent when recommending brands or products: https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/
- OpenAI API documentation, Web search: https://platform.openai.com/docs/guides/tools-web-search
- Google Search Central, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features
- Maya, Methodology: https://withmaya.ai/methodology
- Maya, Trust: https://withmaya.ai/trust
- Maya, How to Measure AI Visibility: The 7 Metrics That Matter: https://withmaya.ai/blog/how-to-measure-ai-visibility-7-metrics
- Maya, 13 Things That Changed in ChatGPT in 2 Months: https://withmaya.ai/blog/13-things-that-changed-in-chatgpt
- Wikipedia, Binomial proportion confidence interval (Wilson score interval): https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

GEO researcher at Maya. Works on brand representation in AI answers and how agencies can scale AI visibility for their clients.