For the past two years I’ve been banging on about the many problems with prompt tracking. The usual defence: “there’s nothing better”.
Solid.
But prompts are not keywords. And answers are not islands.
People don’t prompt in handy context-stuffed strings you can drop into boxes with a neat little “volume” label.
And even if they did, the answer ChatGPT spits out to Paul the plumber from Portland might look completely different to the response it hands over to Mary the marketing manager from Michigan.
What seems like the exact same input (a prompt) can produce two completely (and I mean radically) different outputs.
Because guess what? That prompt isn’t the whole input. It’s only part of the input, and arguably, not even the most interesting bit.
A prompt belongs to a conversation. That conversation belongs to a person. And that person has baggage. Baggage that travels first class with every message.
You know, a little thing called… CONTEXT.
Which means that tracking outputs in lab conditions with zero user context gives you a window into a world that simply doesn’t exist.
All well and good. Maybe (despite my objections) there’s some value to seeing some sort of “baseline”.
But the truth is this: to actually understand where you stand in AI search (and what to do about it) you have to work forwards, not backwards.
And when you do, you discover something rather awkward for the prompt-tracking industrial complex:
Their entire purpose is to track answers. And yet (spoiler) they’re completely blind to most of them.
ChatGPT isn’t generating one answer — it’s generating a cascade of answers. The first one shapes what happens next, and your prompt tracker is completely blind to it.
Alright kids, gather round.
Everyone’s been carping-on about ChatGPT increasing its site:somedomain.com fan-out searches.
But they never really attempt to explain why it’s doing that; other than perhaps OpenAI attempting to reduce spam etc. And that’s definitely true, but it’s not the full story.
What’s really happening is this: ChatGPT is validating its own opinion.
I kind of already spoke about this at length in February in my guide to AI search, although at the time that was more focused on Google AI Mode. And in the interim, it seems Sam’s been copying Google’s homework.
In short, for most (but certainly not all) queries these days ChatGPT kind of knows the answer already, or at least what it thinks is the answer. There are a few reasons for this:
- The gap between training runs has compressed, so the model’s baked-in knowledge can be considerably fresher than it used to be.
- The models are bigger. At the end of the day, when it comes to LLMs, size matters. In general, the bigger the model, the more it “knows”.
- As unintuitive as it sounds, starting with an answer (not the kind the prompt trackers track) makes for better retrieval.
So when you* ask ChatGPT “which CRM should I buy?” it already knows (roughly) what it’s going to tell you. And before it does anything it’s going to generate an answer. But an answer that you (the user that’s trusting ChatGPT with your CRM buying advice) will never actually see.
This might happen in its reasoning trace. A smaller model might generate it separately. But we can afford to be a little hand-wavey here because the exact mechanics don’t particularly matter.
What’s important is that behind-the-scenes we have a draft answer to the user’s query pretty much straight away. A draft answer that’s going to shape everything that happens next.
It might not be exact. It might not include the final brands in the final order. But it’s going to have the right shape (think headings, structure). And it’s going to be packed with the entities, terms, and concepts that ChatGPT believes a good answer to this question should contain.
The draft answer becomes a research plan. Its claims and information gaps become the fan-outs.
So yeah, as I already said, these days the query fan-outs are mainly about collecting evidence that supports the model’s priors (its “opinion”).
Can the model change its mind before final generation? Sure. We still see wide retrieval in the mix. Maybe the consensus has changed since its knowledge cutoff. But you’re going to have to:
- a) get retrieved,
- b) survive selection and filtering,
- c) make a compelling case
And that doesn’t mean ranking yourself at number 1 on your own listicle. These days that’s becoming a massive self-own, and if anything you’re probably helping to reinforce the category leaders.
Here’s a brief example of how an initial draft answer (we’ll keep it short) might be decomposed into fan-outs:
(I have no idea on HubSpot’s pricing or Pipedrive’s onboarding… remember, example)
From this, we might get something like:
Best crm system small team 2026 reviews (wide - no specific brands)HubSpot vs pipedrive comparison 2026HubSpot alternatives 2026HubSpot reviews ratings 2026 -site:hubspot.com -site:reddit.comsite:hubspot.com pricing plans 2026site:pipedrive.com reviews onboardingsite:g2.com hubspot pipedrive reviews ratings 2026
Breaking that down, we have a wide general query (there may be several), then after that the fan-outs are basically trying to validate, refine, or challenge the draft answer.
The model just wants to know it was right (or wasn’t stupidly wrong). Then it can put a cute citation badge on the text that agrees with it most.
But how does it find that text?
By generating… MORE answers.
It’s answers all the way down.
Before kicking off this section, I need to preface by saying that this is likely just one of OpenAI’s scoring and selection mechanisms. They almost certainly use reranking, lexical search, pre-selection, domain whitelists/blacklists, and other internal heuristics that they’ve been refining. I’d also bet the exact scoring method changes query-by-query. Some queries suit semantic search, while others are better with BM25 style precision.
But I think it’s a given that embeddings are heavily involved. And when they are, I’d propose they’re using a form of HyDE. Why? Because it just makes for better retrieval.
HyDE in simple terms: embed the question and you’re representing the semantics of the question. Embed a hypothetical answer and you’re representing the kind of evidence you actually want to retrieve.
Here’s what I suspect happens:
Each of the fan-outs gets its own hypothetical answer-shaped target (HyDE).
For the hubspot pricing fan-out (number 5 above) that might look like:
The numbers and plans here are completely made up. But that doesn’t matter. What matters is that this looks like the kind of words you’d expect to see in a passage that contains pricing information (specific figures, named plans etc).
These HyDE answers might be generated at the same time as the fan-out queries. Or they might be generated while the live searches are pulling results. Again, doesn’t particularly matter.
What’s important is that once we have a bunch of candidate snippets (either from initial search, or after a pre-filter pass) for each fan-out query we have an answer-shaped embedding to score them against.
Grab a bunch of snippets that are most likely to provide the evidence we need. Dump the rest. Rerank them again maybe. Pass them to the model. Model picks the winners and writes the final answer.
Done.
But wait a second… hang about, we missed something. Something… kind of important.
Remember that little asterisk next to you earlier? Time to cash it in.
ALL of the above changes depending on who’s asking.
So we want the model’s pre-retrieval opinion huh? At this point you might be thinking, great, I’ll just fire up the OpenAI API and ask “which CRM should I buy?”, see what it says.
But it doesn’t work does it.
To do the right research, and generate the right answer, the model needs to know who “I” is. Is it a CMO? Is it a founder? Is it your mother’s goldfish?
How does it know? Well… obvious answer is obvious.
In a stateful AI product, the model doesn’t just get the user’s prompt. It’s also going to get relevant context about the user, the current conversation, and information carried forward from previous conversations (think “memories”).
Prompt trackers are well aware of this problem. And many of them try to solve it by injecting the magna carta into a single turn:
Does it work? Well, I mean… maybe to an extent. But is this how you talk to ChatGPT? Or do you just, you know, ask it a question because it already knows who you are?
All of this user context, and I mean all of it, affects everything in the chain.
Change the user, you change the draft answer. Change the draft answer, you change the fan-outs. Change the fan-outs, you change what gets retrieved. Change the retrieval targets, you change which passages survive.
And that’s just the user. We haven’t even got to the conversation yet.
So we’re stuck with trying to simulate it all through synthetic mega prompts.
And we can see the fan-outs in our prompt tracking dashboard of choice, but we’re left guessing why those fan-outs existed in the first place.
Or are we?
What if… we worked forwards?
Do they have a point? Kind of.
If you scrape an answer from the real ChatGPT interface, then yeah, that answer is ‘real’. It happened, in the real user interface, of a real AI search product.
What it doesn’t tell you is whether that answer represents what an actual user would see.
And where the argument starts drifting into strawman territory is the assumption that anything using an API must simply be firing naked prompts at a bare model with no harness, state, retrieval, or orchestration around it.
Some probably are.
I don’t particularly care, because I think prompt tracking is a broken model either way.
But the reason I bring this up is that if we want to make all of this observable, if we want to see how answers (all answers, not just the final one) change based on the user, how conversation context carries forward, why particular fan-outs exist, and how sources are filtered at each stage, then we’re going to have to simulate the mechanics.
Because obviously none of that is sitting there waiting for us in the final UI response.
And most of it isn’t sitting neatly in the network logs either.
If I were going to model this properly, I’d do something like this…
1. I’d start with a person, not a prompt
First, I’d create a persistent user. Let’s call that user a persona. You know, like the ones we’ve been building to model buyer journeys for decades.
But that user wouldn’t be a 400-word persona stuffed into every prompt. It would be an actual stateful profile that sits outside the current message and can be drawn on when it’s relevant.
Their role. Their location. Their goals. Their constraints. Their preferences. Their history. Whatever actually matters.
Then I’d leave the prompt alone.
“Which CRM should I buy?” stays “Which CRM should I buy?”
The context belongs to the user, not the prompt.
2. I’d let the conversation carry its own context
Next, I’d stop treating every question as if it happened in a vacuum.
If the first question is:
“What CRM options should I look at?”
and the next is:
“Which one would be best for our team?”
the second question only makes sense because of the first answer, the conversation around it, and what the system already knows about the user.
So I’d carry all of that forward.
And if the conversation branches, I’d let each branch inherit what came before and then evolve independently.
That way I’d be modelling an actual journey rather than sticking “Awareness”, “Evaluation”, and “Conversion” labels on three unrelated prompts and calling it a customer journey.
You know… conversations.
3. I’d capture the answer before search
Before retrieval, I’d capture the model’s initial position.
- What does it already think?
- Which brands are already present?
- What claims is it preparing to make?
- What does it think a useful answer should contain?
That would give me the hidden draft answer we talked about earlier. The one that shapes everything that happens next.
And suddenly one of the most interesting questions becomes:
Was my brand missing before retrieval even started?
Because that’s a very different problem from being retrieved and then losing.
4. I’d turn that answer into a research plan
Then I’d decompose the draft answer into the things the system needs to verify, update, or challenge.
That gives me the fan-outs.
But instead of merely showing:
best CRM for small teams 2026
I’d keep the relationship between the fan-out and the thing that caused it.
Now I’d be able to see not only what the system searched for, but why it searched for it.
5. I’d give every fan-out an evidence target
For each research task, I’d generate the kind of answer-shaped passage the system is trying to find.
- Pricing information should look like pricing information.
- A comparison should contain entities, differences, advantages, limitations, and qualifiers.
- A review query should pull back evidence that looks like review evidence.
Then I’d retrieve candidate passages and score them against that information need.
6. I’d keep the losers
I’d keep the full candidate trail.
- The pages that were retrieved.
- The snippets that scored well.
- The ones that scored badly.
- The ones that survived another pass.
- The ones that got thrown away.
Because a page being found and a page being used are two completely different things.
Now instead of:
“Competitor X got cited and we didn’t.”
I’d be able to ask:
- Did we never get retrieved?
- Did we get retrieved but lose on relevance?
- Did our evidence make the shortlist but disappear before grounding?
- Were we used but never surfaced in the final answer?
That’s much more actionable.
I’d also probably extract entities, n-grams, and semantic triples from the search results and whack them into a knowledge graph. Then I could inspect the concepts, entities, and relationships being reinforced across the retrieved evidence.
7. I’d then get a model to write the final answer
Only after all of that do I care about the answer the user actually sees.
I’d take the surviving evidence, ground the response, attach the citations, and generate the final answer.
That’s the bit prompt trackers start with.
In this model, it’s the endpoint.
And because I kept everything that happened before it, I could finally stop staring at an answer and trying to guess backwards.
I could follow the chain forwards.
8. Then I’d stop obsessively tracking it
Once I’d built a sensible set of personas and mapped out the buyer journeys that actually matter, I wouldn’t feel the need to rerun the same questions 50 times a day and stare at a line going up, down, and sideways on a dashboard.
I’d run the journeys once a month.
Or when I’d made meaningful changes to the site.
Or when something important changed in the market, the models, the retrieval systems, or the competitive landscape.
Then I could compare the whole chain.
- Did the model’s starting opinion change?
- Did the research path change?
- Did different sources get retrieved?
- Did my page start surviving selection?
- Did the final recommendation move?
I think that's a lot more useful that “AI visibility went from 48.6 to 47.3 today.”
There’s only one slight problem with this methodology.
Figuring all this out and actually building it would take forever.
Not quite.
Around two months as it goes (and 30ish years of search experience).
Introducing QueryBurst Answer Engine
Yeah, I already built it.
Persistent personas...
User and conversation context that travels with the message, not stuffed inside it...
The model’s answer before retrieval starts…
The fan-out, the evidence target, and the passages that survived…
What got retrieved. What survived. What was grounded. What got cited…
Actual buyer journeys with branching conversations...
And when the conversation branches, the advice can branch with it… (HubSpot gets a downgrade on the "cancellation" path)
The whole conversation...
You know, the thing the prompt trackers told us that people have with AI, then tried to compress into a single prompt? Yeah. That.
Everything inspectable. Everything observable. The whole journey. The whole story.
Prompt trackers start with the answer and try to work backwards. Answer Engine starts with the person and works forwards.
QueryBurst Answer Engine is live now.
