Why multi-turn conversations measure something different from isolated prompts

Isolated prompts and multi-turn conversations are not two variants of the same test. The first design measures the response to a controlled input. The second measures a trajectory: how the response changes after clarifications, constraints, intermediate recommendations, and information introduced by the model itself.

The difference matters to a brand's visibility. A company may be missing from the first answer and may appear after the user specifies the budget. It can be mentioned early, but removed when technical requirements arise. Or it can stay in the conversation without being recommended at the end.

If we lump all these events into a single percentage, we miss the very behavior we are trying to understand.

What an isolated prompt measures

An isolated prompt starts in a clean session with no previous messages from the same test. For example: "Which invoicing solutions are suitable for an online store in Romania?"

This design is useful when we want to compare answers to the same question under the same conditions. We can vary the language, wording, engine, interface or timing in a controlled manner. The observation is relatively easy to classify: brand present or absent, cited or not cited, recommended or just mentioned.

The main advantage is comparability. The downside is that real users rarely compress every criterion into one perfectly worded request. A prompt can measure initial discovery, but it does not necessarily show how the selection process develops.

What a multi-turn conversation measures

In a multi-turn conversation, the response on the third turn depends on the preceding messages and answers. The official documentation for the OpenAI Conversations API describes preserving state between calls, while the Google Interactions API can use the identifier of a previous interaction to continue the context.

Implementation differs between systems. In some cases, the history is sent again; in others, state is managed by the server. For an editorial audit, the important takeaway is simple: we must record how the history was preserved. The “multi-turn” label does not describe the test conditions by itself.

Conversation measures things that the isolated prompt cannot observe: time of first appearance, brand persistence, reaction to a new constraint, recovery from misinformation, and conversion of a mention to a recommendation.

A Romanian user journey in five turns

Assume a person who runs an online store and is looking for an invoicing solution:

1. Discovery: “What type of invoicing solution is right for me?”
2. Context: "I have about 2,000 orders per month and I sell in Romania."
3. Integration: "I need to connect with the store platform and courier company."
4. Risk: "What should I check regarding e-Invoice, support and data export?"
5. Decision: "Give me a short list and explain why you kept each option."

On the first turn, we can measure the proposed category and criteria. A list of providers may appear on the second turn. On the third, the list may narrow. On the fifth, we record the final recommendation and its justification.

It wouldn't be fair to turn each step into a separate prompt and pretend we've reproduced the same conversation. The “I also need integration with a courier” message is ambiguous without the preceding invoicing and online-store context.

When brand input contaminates the test

If the evaluator asks “What do you think of Brand X?”, the model did not discover the brand organically. The test may measure perception, factual accuracy or comparison, but it should not enter the organic visibility denominator.

There is an important distinction, however. If the model introduces Brand X in the second turn, the simulated user can then ask "Which of the mentioned options integrates with my platform?". The brand remains in context without the reviewer injecting it before the initial appearance.

For each turn, we keep a provenance field:

• user-seeded — the name appears in the user message before the model enters it;
• model-introduced — the model first introduces it into the conversation;
• context-carried — appears later because it was already part of history.

Only the second state demonstrates organic discovery in that trajectory.

Fixed script, adaptive user or hybrid protocol?

A fixed script sends the same replies regardless of the response. It's reproducible, but it can get silly: the user asks for details about a list that the model didn't provide.

A fully adaptive simulated user responds naturally to each output. It produces plausible conversations, but adds a new source of variation. Two runs can follow different branches and we are no longer comparing the exact same test.

A hybrid protocol keeps the persona, objective, criteria and stopping conditions fixed. Branching is allowed only within a declared set. For example:

• if the model asks for the budget, the user responds with the default range;
• if it gives a list, follow the integration question;
• if the model refuses or fails to understand the task twice, the conversation stops;
• if the model recommends an option, the user asks for evidence and limitations.

This way we keep realism without letting the evaluator arbitrarily steer the conversation towards a desired outcome.

Correct units and denominators

A conversation with five turns should not automatically be counted as five conversations. We can report separately:

• conversation visibility = conversations with at least one organic appearance ÷ all eligible conversations completed;
• turn visibility = eligible turns with an appearance ÷ all eligible turns;
• first organic mention turn = number of the turn on which the model first introduces the brand;
• persistence = subsequent eligible turns in which the brand remains a candidate ÷ all eligible turns after first appearance;
• recommendation conversion = conversations where an organic occurrence becomes a final recommendation ÷ conversations with organic occurrence.

These formulas extend the convention explained in The denominator changes the result: the hidden trap in AI KPIs.

An error can propagate between turns

A wrong answer at the beginning can influence the entire trajectory. The model may mistakenly assume an integration, and the simulated user may proceed based on it. The end result is not a series of independent observations.

MT-Eval analyzes patterns such as recollection, expansion, refinement and follow-up, and reports degradation in multi-turn contexts for some tasks. This does not tell us which brands a particular engine recommends, but it supports separating conversation-level evaluation from single-turn scores.

When classifying, we mark the first turn on which the error occurs and the dependent turns. We do not attribute five independent errors to one repeated false premise.

Replays, clean sessions and comparable conditions

Each replay starts in a clean session. Otherwise, the previous conversation may contaminate the next observation. We log the available engine and version, interface or API, date, language, geography, settings, state mechanism, and any enabled tools.

Runs must be distributed over time if we want to estimate stability, not merely repeated within one minute. We also report failures: missing response, context limit, refusal, interruption or an unusable branch.

NIST differentiates performance conditional on a fixed benchmark from generalization to other similar examples. This distinction is crucial: a set of 20 conversations describes the tested set, not all possible user behavior.

What such a test cannot demonstrate

A simulated conversation is not a direct estimate of actual market share, nor proof that users will buy the recommended product. We don't know the full distribution of people, wording, and intent from real use, and interfaces can personalize or alter responses in ways the test doesn't detect.

Even repeated appearance in a conversation set does not demonstrate why the brand was chosen. We can analyze the justification and the sources shown, but they do not expose all the internal processes of the system. The link between an editorial intervention and subsequent change should be treated as a hypothesis until there is a comparable before/after design and adequate controls.

Therefore, the report must distinguish three levels: the raw observation, the classification made by the evaluator and the business interpretation. "The brand came up in 12 out of 30 conversations tested" is one observation. "It is well known" is an interpretation that requires further evidence.

These limits do not make the test useless. They define it as a diagnostic tool: one that shows where information coverage is worth checking, which claims are uncertain and which scenarios need further investigation.

LLM-as-judge does not replace validation

A model can help label mentions, recommendations, and justifications, but the automated evaluator must be tested on a human-labeled sample. The rubric must separately define mention, listing, conditional recommendation and final recommendation.

If the same model generates the simulated user and judges the result, correlated preferences may occur. We retain the raw output, rubric version, and final decision. Ambiguous cases go under review, not forced into a favorable category.

Minimum specification for a multi-turn test

1. explicit persona and objective;
2. context, language and geography;
3. fixed criteria and information that the user can disclose;
4. branching rules;
5. stopping conditions and maximum number of turns;
6. rule for user-entered brand names;
7. engine, interface, date and state mechanism;
8. number of retries and clean sessions;
9. turn-level and conversation-level metrics;
10. rubric, human validation and transcript retention.

This protocol follows the general definition in What AI visibility is and how to measure it correctly, but it does not assume that any product already implements every field or integration described.

Conclusion

An isolated prompt answers the question “What does the engine produce for this input?” A multi-turn conversation answers “How does the selection evolve as context and criteria accumulate?”

We need both. Isolated prompts provide control and diagnostics. Multi-turn conversations approximate the trajectory, persistence and conversion of an appearance into a recommendation. Their scores, however, must not be mixed.

When we preserve brand provenance, branches, history, stopping conditions, and denominators, the conversation becomes an auditable experiment. Without these elements, only a convincing demonstration remains.

Sources

• OpenAI API — Conversations
• Google AI for Developers — Interactions API
• Google AI for Developers — Multi-turn text generation
• NIST — Expanding the AI Evaluation Toolbox with Statistical Models
• MT-Eval — EMNLP 2024
• MultiChallenge — Findings of ACL 2025

Last source check: July 31, 2026. Journey examples and formulas are editorial methodology, not results of an AYSA benchmark and not claims about already released features.