Why Real AI User Data Is Not Automatically Representative of a Market
QueryBurst does not treat real-user conversation data as automatically representative of a market. Real-user provenance establishes where the data came from — it does not establish that a number calculated from those conversations represents a market.
Real-user data can be extremely valuable for research and discovery: it reveals the language people actually use with AI systems, the questions they ask and the behaviours that emerge over time. QueryBurst does not dispute that value.
Our position is narrower: real-user provenance is not the same thing as statistical representativeness. Those are different claims, and they should not be treated as interchangeable.
Real-user tells us where the data came from
A dataset collected from real AI users has an obvious advantage over a purely synthetic dataset: the source behaviour actually happened. But before that behaviour can be interpreted as representative of a wider population, other questions become important.
Who is in the panel? Who is not? Are heavy AI users represented more often than occasional users? Are particular devices, countries, demographics or platforms easier to observe? Do people who opt into behavioural panels differ systematically from people who do not? How are those differences corrected? How much uncertainty remains after correction?
None of those questions make panel data invalid. Most market research relies on samples rather than observing an entire population, and weighting, modelling and extrapolation are normal ways of turning observations from a sample into estimates about a larger group.
QueryBurst’s position is that the distinction matters:
Real-user provenance tells us that the observations are genuine. Representativeness tells us how confidently those observations can be generalised.
One does not automatically establish the other.
A large corpus is not the sample size of every claim
QueryBurst applies the same distinction to dataset size.
A company may legitimately possess hundreds of millions or billions of real-user conversations. That can be an extraordinary research asset. But the size of the entire corpus does not tell us how many observations support a particular conclusion.
A specific market-level claim may be produced only after filtering the underlying corpus by date, platform, country, category, language, relevance, intent or some other condition. The resulting records may then be deduplicated, clustered or compressed into a smaller set of representative questions.
The final measurement may be based on something narrower again.
So the relevant question is not simply:
How large is the dataset?
It is:
What evidence actually supports this particular claim?
QueryBurst’s September 2026 research into Profound’s conversation dataset, Prompt Volumes and Index methodology provides a worked example. Profound publicly describes a corpus of more than 1.9 billion real-user conversations. Its published Index methodology does not calculate individual brand rankings by counting brand appearances across all of those conversations. The corpus is used upstream for retrieval, filtering, clustering and representative-prompt selection; the resulting prompt set is then executed afresh against AI systems to produce the brand measurement.
That is not necessarily a problem with the methodology. It does mean that:
“Powered by 1.9 billion conversations” is not the same statement as “this result is based on 1.9 billion observations.”
The first describes the research foundation available to the system.
The second would describe the evidence supporting the individual estimate.
They should not be treated as interchangeable.
Prompt, conversation, user and demand event are different units
QueryBurst considers this an underexamined complication with conversational data: deciding what one observation represents.
A prompt is not necessarily a conversation.
A conversation is not necessarily a user.
A user is not necessarily a purchase journey.
And a new prompt is not necessarily a new unit of demand.
Consider a customer having this conversation with an AI assistant:
- “I need a CRM.”
- “Something for a small team.”
- “Salesforce is too expensive.”
- “What about HubSpot?”
- “Anything cheaper?”
- “Which one would you choose?”
There are six user turns.
There is probably one underlying decision.
Counting those six turns may be perfectly legitimate for some forms of analysis. Treating them as six independent expressions of market demand would be a different claim.
QueryBurst considers this increasingly important as AI usage becomes more conversational.
Later turns such as “Which one would you choose?” or “What about for a small team?” can be highly meaningful inside a conversation while containing very little standalone information about the user’s intent.
The conversation state is carrying information that the individual prompt no longer contains.
The more conversational AI becomes, the less complete the individual turn becomes as a unit of analysis.
Any market-level interpretation of conversational data therefore depends partly on what the measurement unit is and how repeated turns, persistent threads and prolific users are handled.
Those questions do not prove that a resulting estimate is wrong. They determine what the estimate actually represents.
What QueryBurst does instead
QueryBurst does not run prompt panels, does not calculate AI visibility scores from prompt-level observations, and does not track brand mentions across isolated AI responses. QueryBurst is built for investigation rather than tracking: users model multi-turn customer journeys through AI search and inspect where a brand enters, survives or disappears across the full answer process.
This page addresses a different question: whether real-user conversation data, collected by companies that do use those methods, automatically establishes the representativeness of the resulting market-level claims.
Modelling is not the problem
QueryBurst does not object to statistical modelling.
A panel almost always has to be modelled if the goal is to estimate behaviour outside the observed sample.
Demographic correction, geographic weighting, population scaling and other adjustment techniques can all be completely appropriate.
The question is not whether modelling happened.
The question is whether the resulting claim is scoped to what that model can reasonably establish.
A modelled estimate should not quietly become synonymous with an observed count.
A panel should not quietly become synonymous with a population.
And the size of the source corpus should not quietly become synonymous with the statistical support behind every number produced from it.
Real-user data can still be extremely useful
QueryBurst considers real conversational data particularly useful for research and discovery.
Real conversations can reveal language researchers would not have anticipated, emerging topics, unusual constraints, changing behaviour and patterns that a hand-written prompt list may simply miss. That is valuable precisely because the data is messy.
Real people do not naturally organise their questions into neat benchmark panels.
They change their minds. They refer back to earlier messages. They name brands. They compare products. They introduce new constraints. They ask vague follow-ups. They move between informational and commercial intent inside the same conversation.
That mess is part of the behaviour being studied.
Compressing it into a benchmark may be useful for a particular measurement task.
But the benchmark and the original behaviour are no longer the same thing.
A benchmark can be valid without being a market
QueryBurst draws a further distinction when real-user data is used to construct standardised benchmarks.
A benchmark may deliberately remove some kinds of behaviour in order to make comparisons fairer. For example, a benchmark intended to measure spontaneous brand recommendation might reasonably exclude questions that already name a brand. That can be a perfectly defensible design. But it changes what is being measured.
The resulting number describes performance within a constructed brand-agnostic benchmark. It does not automatically describe brand visibility across the natural mixture of branded, comparative, product-specific and open-ended conversations occurring in the wider market.
QueryBurst’s September 2026 review of Profound’s Index methodology provides a concrete example. Profound’s current Index explicitly filters its final prompt set toward open-ended, brand-agnostic questions and removes prompts naming specific brands. That may be appropriate for the benchmark Profound wants to construct. It also means the benchmark is measuring a deliberately selected subset of buyer behaviour.
QueryBurst’s view is that the important question is therefore not whether the benchmark is “real” or “synthetic.” It is:
What population of behaviour is this benchmark intended to represent?
Representativeness has to be demonstrated at the level of the claim
This is ultimately QueryBurst’s position.
We do not assume a dataset is representative simply because its source records came from real people.
We do not assume a billion-record corpus means a narrow market estimate has a billion-record sample.
We do not assume a prompt, a conversation, a user and a demand event are interchangeable units.
And we do not assume a standardised benchmark represents the full distribution of market behaviour simply because real-user data contributed to its construction.
Before treating a panel-derived number as a market-level estimate, we want enough information to understand the sampling frame, the observation unit, the weighting or modelling process, the effective evidence behind the specific claim and the uncertainty around the resulting estimate.
That does not mean every commercial methodology must be completely reproducible from the outside.
It means the scope of the claim should be clear enough to understand what the number represents.
Our position is simple:
Real-user data establishes provenance. It does not establish representativeness.
And:
The corpus size is not the sample size of the claim.
Real-user data can still be excellent evidence.
It is not, by itself, proof of a market.
For a detailed worked example, see QueryBurst’s September 2026 research into Profound’s 1.9 billion conversation dataset, Prompt Volumes and Index methodology, where we document the company’s published methodology, source claims and unresolved questions using first-party material.