Measuring AI Visibility: Method, Metrics and What Providers Must Prove

The board wants figures before it releases budget: how present is our company in AI answers, and where does the competition stand? So someone opens ChatGPT, types the obvious question and gets a list of five providers. Their own company is not on it. Screenshot, slide, alarm. Two days later a colleague asks the same question and the company sits in second place.

Both measurements are worthless. Not because anyone made a mistake, but because a single query carries no information with these systems.

Key takeaway: AI systems answer the same question differently, and this is not random noise but measurable and systematic. A Conductor study covering 14,000 queries shows that for purchase-intent questions, only 40 percent of the brands named overlap between two runs of the same question. A defensible measurement therefore needs a fixed question list, several repetitions per question, and an analysis that states its uncertainty. Since 3 August 2026 there is also a yardstick from the industry body: the IAB's Measuring Visibility in the AI Era separates directional from decision-grade measurement and sets out what measurement vendors must disclose. Anyone deriving a percentage from five queries is reporting randomness with decimal places.

Why does the same question produce a different answer every time?

Because language models do not look answers up. They calculate probabilities word by word. Tiny numerical differences are enough for a different word to win at a fork, and from there the answer runs in another direction. With provider lists, that is exactly what decides who gets named.

What is notable is that the variation persists even when you set the model temperature to zero. Thinking Machines Lab showed why in September 2025: the compute kernels on the graphics card produce different results depending on how many requests are being processed at the same time. Your request does not change, but its neighbourhood in the batch does, and with it the rounding in the last decimal place. Only after the team made the affected kernels independent of batch size were 1,000 consecutive runs bit-for-bit identical.

The practical consequence: this variation is not a quirk of individual vendors, and it does not disappear once you find the right switch. It belongs to the method.

Conductor measured its magnitude systematically in July 2026: 70 questions, 50 independent runs each, four systems (ChatGPT, Perplexity, Claude and Gemini), ten industries, 14,000 queries in total. The result contradicts the common assumption that the answers are simply arbitrary.

How to read this: for purchase-intent questions, two answers to the same question overlap in only 40 percent of the brands they name. Precisely where it matters most commercially, the spread is widest. Comparison questions are the most stable at 63 percent, and the brand named first stays the same in 91 percent of runs. Educational questions show a pattern of their own: the set of brands is fairly stable at 60 percent, but which of them comes first changes in 70 percent of runs.

A second finding from the same study: Gemini names 9.2 brands per answer on average, ChatGPT only 5. Anyone merging both systems into one metric is comparing unlike things.

Context worth knowing: Conductor sells software for search and AI visibility and therefore has an interest in the topic being seen as demanding. That said, the methodology is disclosed and reproducible, and the order of magnitude matches what we see in our own measurement series. The figures are usable, but their origin should be known.

Which metrics actually exist?

AI visibility is not a single number but a bundle of six. They answer different questions, and most internal debates derail because two people are talking about different ones.

The most important of the six is factual accuracy, and it is the only one that cannot be automated. A mention containing false claims can be commercially worse than no mention at all. How widespread this is was shown by a survey from the agency maxonline in March and April 2026: across 150 mid-sized companies in eleven industries in the German-speaking region, tested with more than 450 questions, ChatGPT returned false or invented details for 96 percent of queries about managing directors. The same caveat applies here: the study comes from an agency selling GEO services. The finding does match what anyone can verify by querying their own management team.

What you can do about false claims is covered in the article on false and vague AI claims about your company. In short: there is no reporting route, only the detour via the sources.

What has the IAB standard changed?

It has given these metrics a shared vocabulary. The Interactive Advertising Bureau released Measuring Visibility in the AI Era in New York on 3 August 2026, developed by a working group chaired by Caroline Giegerich, the body's VP of AI.

The trigger is a market failure. More than 20 firms now sell tools for measuring AI visibility, each with its own methodology, and the same brand receives different numbers from different vendors. At the same time, according to the framework, only 16 percent of brands track their visibility in AI answers systematically at all.

The framework sorts every metric into four groups, the 4 P's in the original. They build on one another: first whether a mention happens at all, then how prominent it is, then in what light, then to what effect.

Two distinctions inside this matter more in practice than they first sound. First, the standard separates mention from citation: a brand can be named constantly in answers and still almost never be linked as a source. These are two metrics with two different levers. Second, it separates hallucination rate from factual inaccuracy: in the first case the model invents a connection with no source behind it, in the second there is a real source but the detail drawn from it is wrong. The first is a model problem; the second is one you can fix through source work.

How many repetitions does a defensible figure need?

More than most people expect. If you ask a question five times and are named twice, that is 40 percent on paper. Statistically, the true value sits somewhere between 12 and 77 percent. That range is too wide to support a decision.

The table below shows how uncertainty develops with the number of repetitions, in each case at an observed share of 40 percent. The intervals are calculated using the Wilson method, which is more reliable for small samples than the widely used approximation.

Anyone wanting a mention rate accurate to plus or minus 10 percentage points needs around 80 repetitions of the same question. No monitoring vendor says this gladly, because it makes the effort visible.

From this follows a practical rule that limits effort without becoming dishonest: with small samples, report categories rather than percentages. Four levels are enough for any board presentation.

1. Never named: in not a single run
2. Occasionally: in fewer than half the runs
3. Regularly: in more than half
4. Always: in every run

That keeps the statement defensible, and nobody has to pretend that a move from 38 to 42 percent is a result. For metrics you do want to report as percentages, the floor is 30 observations per cell, as is standard in market research.

When is a figure decision-grade?

When it meets the nine requirements the IAB framework sets for it. The framework distinguishes two quality tiers. Directional means the measurement shows whether things are moving up or down; it serves internal monitoring and competitive awareness, and it is explicitly not enough for a budget decision. Decision-grade means the number can carry an investment decision.

The figure of 50 is a floor, not a target; how many repetitions a specific claim needs is shown in the table in the previous section. The repeatability criterion is the notable one. It does not demand that a measurement be exactly reproducible, which is impossible with language models. It demands that the vendor states in advance how much the values are allowed to move. Without a tolerance band, there is no way to tell an effect from noise.

Not every company needs decision-grade measurement. It exists for budget decisions; anyone without a large sum to allocate gets a long way on directional measurement, provided it is labelled as such.

How do you assemble the question list?

With 25 to 50 questions your customers actually ask, and then those questions stay unchanged. Word for word, over months. The moment you adjust the wording, you are measuring the wording rather than the development of your visibility.

Below 25 questions, individual cases dominate the result; much above 50 and the list stops being maintained without automation. In practice the useful range for a mid-sized company is 30 to 40 questions.

The composition should reflect the intent types that performed with differing stability in the Conductor study. A workable split:

The most common mistake here: almost only brand questions get asked. If you want to know what a prospect experiences who does not yet know your company, you have to ask the problem question, not the company name. Brand questions measure whether the system knows you. Problem questions measure whether it recommends you. For the pipeline, the second one counts.

How do you calculate your share against competitors?

Through your share of all brand mentions. You count every provider named in each answer, sum across all runs, and divide your own mentions by the total. With 40 own mentions among 500 brand mentions overall, your share is 8 percent.

Two rules make the comparison fair. First, evaluate each system separately. Because Gemini names 9.2 brands on average and ChatGPT names 5, a combined figure dilutes the statement. Second, the competitive set is defined in advance and not adjusted afterwards. Otherwise the temptation arises to redefine the field until the numbers look acceptable.

For a board presentation, share of mentions is the most intuitive figure because it reads like market share. It should never stand alone, though, but always alongside the number of observations it rests on.

What does Search Console show that no monitoring tool sees?

Search queries that no human types. There is one measurement channel that costs nothing and appears in no product demo: your own Google Search Console.

In our own data for arkeontech.de, several fully written-out prompts appeared among the normal search terms in July and August 2026. One of them ran to more than 300 characters and ended with an instruction to search online for specialised providers and name specific ones. Strings like this arise when an AI system runs a web search on behalf of a user and the query reaches Google.

The value of these finds is considerable, for two reasons. First, they contain the actual wording of the customer problem, in more detail than any keyword tool provides. Second, next to it stands the position your page was served at. That is a genuine visibility measurement in a generative context, free of charge, in your own account.

How to find them: in the Search Console performance report, scan the queries by length. The prompts sit alphabetically among the short keywords and are easy to miss. Look for queries longer than roughly 80 characters and for full sentences, often with instructions such as research online or name specific providers. How to recognise such machine queries reliably, and what they mean for click-through rates, is covered in the article your customers no longer search, they brief the AI.

One caveat belongs here: these queries are rare, and they capture only the part of AI usage that runs through a Google search. As a complete measurement it does not qualify. As an early warning system and a source for your own question list, it is excellent.

What mistakes do most measurements make?

Six patterns come up repeatedly, and five of them cost nothing but discipline.

Measuring in a logged-in account. Anyone testing in their own ChatGPT account is measuring what the system has stored about them. Saved memories and earlier conversations feed into the answer. Measure logged out or in a temporary chat without history, otherwise you are measuring your own usage history.

Confusing being mentioned with being described correctly. Your company gets named, everyone is pleased, and nobody checks the details in the subordinate clause. Every mention needs a fact check attached.

Phrasing the question slightly differently each time. Even a changed word order can tip the result. The question list gets frozen and versioned.

Deriving a trend from one measurement. A trend needs at least three measurement points at equal intervals and the same methodology across all three.

Measuring through the API and talking about the product. Querying the API measures the model. Using the web interface measures the product, including the web search in front of it and the vendor's system instructions. Both are legitimate, but they are different things. For what a prospect actually sees, the product is what counts.

Keeping no raw data. Without stored answer texts there is no way to establish later why a value jumped. Save every answer in full, with date, system and question.

Measure yourself, buy a tool, or hire an agency?

All three work. They differ in effort and in how much control you keep over the methodology.

The monitoring software market settled in 2026; published prices range from around 29 US dollars a month for simple checks to 295 or 399 US dollars for packages covering several systems and competitive comparison, with enterprise plans quoted individually.

What must a provider disclose?

What it measures, how it measures, and how it handles the data. The IAB framework sets out disclosure obligations in these three groups, and they are the most practical checklist currently available for evaluating a quote, because every item is a question you can ask in a meeting.

Query origin is where quotes differ most visibly. A prompt library invented at a desk measures something other than one derived from actual search behaviour. Both are legitimate, but it has to be stated.

Re-baselining sounds technical and has an immediate consequence for reporting: if a vendor changes its model or its prompt library, the numbers before and after are no longer comparable. If that is not in the report, a decline looks like a failure when in fact only the instrument changed.

Which promises do not hold up technically?

Any guaranteed placement. There is no position one in ChatGPT: language models do not return the same answer to the same question, and this is not a teething problem but part of how they work.

The Semrush Visibility Index 2026 shows how large the variation is even for big brands: over 1,200 brands across 22 industries were examined from January to April 2026 using 126 million queries in ChatGPT, Gemini, Google AI Mode and AI Overviews. Only 36 of those brands appeared among the hundred most-mentioned on every platform in every month. If brands of that size cannot hold their visibility steady, a guarantee to a mid-sized company is not credible.

Three further formulations deserve scrutiny:

We will get you into the AI training data. Nobody can commit to that. Which content enters a training run is decided by the model provider, and the next run is months away. What can actually be influenced is retrieval at runtime, and that takes effect in weeks.

Visibility score 78 out of 100. A composite score without a disclosed formula is packaging, not a metric. Ask which of the four categories it draws on and how they are weighted.

We measure daily across all AI systems. Possible, but expensive. Ask for the query volume per run. Daily measurement with five questions is worth less than weekly measurement with fifty.

How do you evaluate a specific quote?

With six questions you can ask in a first meeting, each with an answer that can be held against the standard.

1. Is the measurement directional or decision-grade? A provider who knows the difference and states their own tier has read the framework. Both answers are legitimate; the first is simply cheaper and carries no budget decision.
2. How many queries sit behind each percentage in the report? Below 50, the number is not decision-grade under the standard.
3. What tolerance band applies? Without one, effect cannot be separated from fluctuation.
4. Where do the questions come from? Invented or drawn from real search behaviour, and if real, from which source.
5. What happens when a platform changes? A provider with a procedure re-baselines and says so.
6. Are mention and citation reported separately? If not, there is no basis for deciding whether to work on source presence or on discoverability.

Two practical consequences follow. Ask to see the prompt library before signing, not after. And insist that the first report establishes a baseline against which later reports are measured. Further questions, aimed less at measurement than at the working relationship, are in our checklist for the first meeting with an AI agency.

How often should you measure?

Monthly for ongoing observation, weekly only while a campaign is running or when the figure has to be decision-grade. Measuring more often produces movement in the chart that is mostly spread rather than development.

The reason lies in how slowly these systems move. If a system answers from its training state, the statement changes at the earliest with the next training cycle, and that takes months. If it searches live, changes to your content take effect considerably faster, but even then weeks pass before this shows up in a mention rate. Why systems react so differently comes down to where they draw their knowledge from, which we described in more detail using the example of an AI agent's knowledge base.

A realistic cycle for a mid-sized company: 30 questions, three systems, five runs per question. That is 450 queries a month and roughly half a working day once the analysis is set up. For the ten most important questions it is worth going to 30 runs quarterly, so that at least those metrics hold up as percentages.

Frequently asked questions

Is it enough to simply ask ChatGPT about my company? No, for two reasons. First, the answer varies considerably between runs; for purchase-intent questions only around 40 percent of the named brands overlap. Second, asking about the company name measures awareness, not recommendation. What matters is the question asked by a prospect who does not yet know you.

How many questions do I need to start? 25 to 50, with a sensible focus at 30 to 40. Below 25, individual cases shape the picture; above 50 the list stops being maintained without automation. More important than the count is that the wording stays unchanged.

What is the IAB measurement standard for AI visibility? Measuring Visibility in the AI Era is a framework published by the Interactive Advertising Bureau on 3 August 2026. It defines a shared vocabulary for visibility metrics in AI answers, sorts them into four categories, distinguishes two quality tiers and sets disclosure obligations for measurement vendors.

What does decision-grade measurement mean? A measurement meeting nine requirements: at least 50 queries, a sufficient sample, coverage of all four query types, weekly or more frequent collection, a defined tolerance band, documented data validation, complete methodology documentation, several platforms and a disclosed aggregation method. It is the precondition for basing a budget decision on a number.

Does a mid-sized company need decision-grade measurement? Only if a sizeable investment decision depends on it. For ongoing monitoring, directional measurement suffices, provided it is honestly labelled. More important than the tier is having the prompt library, the scope and the tolerance band in writing.

Do I have to measure through the API, or is the web interface enough? It depends on the question you want answered. The API delivers the model without a web search in front of it and without the vendor's system instructions; the web interface delivers what a customer actually sees. For visibility in customer contact, the web interface is the right measurement point, even though it is harder to automate.

What is the difference between mention and citation? A mention means the brand name appears in the answer. A citation means your own page is given as a source and linked. Either can occur without the other, the standard tracks them as separate metrics, and they call for different measures.

Can an agency guarantee a mention in ChatGPT? No. Language models are not deterministic and there is no fixed placement. The Semrush Index 2026 shows that of more than 1,200 brands examined, only 36 appeared among the hundred most-mentioned on every platform in every month. A guarantee promises something the technology cannot deliver.

How do I recognise in Search Console that a query came from an AI system? By its length and sentence structure. Queries over roughly 80 characters, written in full sentences and often carrying instructions such as research online or name specific providers, practically never come from a person typing into a search box. They arise when an AI system searches on behalf of a user.

What is a good value for the mention rate? There is no general answer, because it depends on the size of the competitive field. What carries meaning is the comparison with the three to five providers you actually compete against, and the development across several measurement cycles. An absolute target without that context is a number without meaning.

How long until measures show up in the measurement? With systems that search the web live, first changes are possible after a few weeks. With answers from the training state it takes until the next training cycle, so months. Plan for an observation period of at least a quarter before evaluating any measure.

Conclusion

The board's question is legitimate: figures belong on the table before a budget release, not gut feeling. But the obvious answer, asking ChatGPT once, is not a measurement. It is a sample of size one in a system demonstrably known to vary.

A defensible measurement is no great art, but it has requirements: fixed questions, several runs, separate analysis per system, stored raw data, and an honest statement of how certain a figure is. Since the IAB framework, this can also be demanded of providers: a provider does not have to deliver the best number, but it must be able to say how the number came about. Anyone offering a score without a formula and a guarantee without a technical basis does not.

For assessing your own position there is an order: first establish whether the company is mentioned at all, then whether it is described accurately, and only then measure how often. Getting started costs less than the numbers suggest: write down 30 questions, query three systems while logged out, five times per question, save the answers with dates. Half a day. After that you know where you stand, and the second cycle next month shows a direction for the first time.

This article was published on 10 August 2026. On 23 September 2026 the article on the IAB measurement standard from 25 August 2026 was merged into it.

Sources

• Conductor: AI Brand Recommendation Study (17 July 2026): 14,000 queries, 70 questions, 50 runs per cell, four systems
• Thinking Machines Lab: Defeating Nondeterminism in LLM Inference (September 2025)
• Interactive Advertising Bureau: Measuring Visibility in the AI Era, published 3 August 2026, working group chaired by Caroline Giegerich (VP AI, IAB)
• Semrush AI Visibility Index 2026, 126 million US queries from January to April 2026, more than 1,200 brands across 22 industries, four platforms (ChatGPT, Gemini, Google AI Mode, AI Overviews)
• maxonline Marketing hfw GmbH, AI hallucination study for the German-speaking region (March to April 2026): 150 companies, eleven industries, more than 450 questions
• Own calculation of confidence intervals using the Wilson method
• Google Search Console, performance data for arkeontech.de (July and August 2026)