Why Example Order Can Change an AI Answer

Suppose you ask an AI assistant to sort twelve customer messages into three queues: billing, technical support and account access. You include six worked examples so the categories are clear. The results look sensible. Then you move the same examples into a different order, change nothing else, and two messages switch queues.

The examples were not decorative instructions. They became part of the computation. This technique is called in-context learning: a language model infers a task from demonstrations placed inside its prompt, without its underlying parameters being updated. The useful surprise is that a model can adapt from a handful of examples. The inconvenient surprise is that the sequence and position of those examples can affect what it infers.

The practical thesis is narrow: if examples help an AI workflow decide or classify, treat their order as a variable to test, not as harmless formatting. Stability under a few reasonable permutations is not proof of correctness, but instability is evidence that the workflow is leaning on accidental prompt structure.

Examples create a temporary task

A conventional trained classifier learns from many labelled records by adjusting numerical parameters. Few-shot prompting does something different. The model has already been trained. At use time, the prompt contains a small pattern such as:

Message: I was charged twice.
Label: billing

Message: The app closes when I open it.
Label: technical support

Message: My password reset link expired.
Label: account access

The model continues the sequence for a new message. It can infer both the output format and a working boundary between the labels. The original GPT-3 work helped establish that large autoregressive models could perform varied tasks this way, with examples supplied in the context rather than through task-specific parameter updates.

But a prompt is a sequence, not a bag of independent facts. Every demonstration occupies a position. Each token is processed in relation to earlier tokens, and the final query arrives after a particular local history. Labels may also have unequal prior associations from training. Researchers have found effects linked to recency, common label tokens, example selection, template wording and the distribution of labels. There is no single universal explanation for every model and task; the safe mental model is that demonstrations jointly define the temporary task, while their layout can tilt it.

How large can the order effect be?

In an ACL 2022 study of few-shot text classification, Yao Lu and colleagues reported that permutations of the same demonstrations could move performance from near random guessing to near state of the art in their experimental settings. A good order for one model did not reliably transfer to another. Their prompt-selection method produced a 13% relative improvement across eleven established classification tasks for the GPT-family models they tested.

An earlier ICML 2021 study found instability from example choice, format and order in GPT-2 and GPT-3. It connected some of the behaviour to preferences for answers placed near the end of the prompt or common in pre-training data. A contextual calibration method improved accuracy by as much as 30 percentage points in the reported tasks. These are results from particular older models and benchmarks, not a forecast for every current assistant.

The issue has not vanished from more recent research. An EMNLP 2025 paper tested ten models from four open model families across classification, question answering, summarisation and reasoning. In that setup, placing demonstrations at the start was generally the most stable and accurate arrangement, with gains of up to six points; moving them to the end of the user message flipped more than 30% of predictions in some question-answering tests without improving correctness.

The counterargument: examples can also stabilise

It would be wrong to conclude that demonstrations make prompts unreliable. Often they remove ambiguity by showing the desired format, level of detail and category boundary. A 2024 prompt-sensitivity study found that adding even one exemplar almost always reduced sensitivity in the open models and tasks it examined, although other prompt variations still mattered.

Modern instruction-tuned models may also be less sensitive than early GPT-3 on some tasks, and an open-ended writing request is not the same as a fixed-label benchmark. The magnitude of order effects depends on the model, prompt, examples, task and evaluation. That variability strengthens the practical point: a reassuring average from another benchmark cannot establish stability in a local workflow.

Nor is invariance always desirable. If one rearrangement makes an ambiguous boundary visible, the changed output may reveal that the examples genuinely imply different rules. The goal is not to force identical answers. It is to learn whether the intended rule or an accidental sequence is driving them.

Run a prompt permutation test

Choose one low-risk, repeatable classification task with an answer that a person can review: routing public enquiries, tagging non-confidential notes or sorting a reading list. Prepare twelve test items and six labelled demonstrations. Write a short rule for each label before asking the model, so the human standard does not move after the outputs appear.

Run four versions in fresh conversations with the same model and settings:

1. an alternating sequence of labels;
2. the exact reverse order;
3. examples grouped by label;
4. the original sequence with the last example moved to the first position.

Do not change wording, examples or test items between runs. If the interface exposes temperature or a random seed, keep them fixed; if it does not, repeat each order twice so ordinary sampling variation is not confused with an order effect. Save the predicted labels, not the conversation’s hidden reasoning.

For each test item, count how many of the four orders produce the majority label. Then calculate a simple flip rate: the number of items whose label changes at least once, divided by twelve. Review every flipped item against the written category rules. A high flip rate means the prompt is not yet a dependable classifier. It does not tell you which order is correct.

Improvement can take several forms: make the label definitions explicit, replace ambiguous demonstrations, balance the number of examples per class, separate the instruction from the examples, or use a conventional tested classifier when the task is stable and high-volume. For consequential decisions, prompt permutation is only a stress test; it cannot replace representative validation data, error analysis and an accountable review process.

Test the temporary training set

A few examples can be one of the most efficient ways to show an AI what useful work looks like. That convenience makes them easy to mistake for illustrations. They are closer to a temporary training set—small, powerful and capable of carrying accidental bias.

The next useful step is therefore not to search for one magical example order. Keep the examples fixed, permute them, and inspect what moves. If reasonable rearrangements change the decision, the prompt has revealed an uncertainty that the polished first answer concealed.