Context Order Bias: Why This LLM Only Sold Apples and Oranges

The problem: two fruits never get sold

A conversational AI was built to recommend fruit to customers. Five products in the catalogue. Simple system prompt with product descriptions and a set of behavioural rules. The kind of setup that looks right on first inspection.

We ran it 10 times with the same user input, “hello, I’d like some fruit recommendations”, and counted which fruits appeared in the responses.

Strawberries and watermelons were never recommended. Not once across 10 conversations.

The setup

The application used a standard chat completion pattern, a system prompt with product descriptions, behavioural rules, and a user message:

system_prompt_input = '''
You are a helpful assistant that always provides concise
and relevant answers to user queries. You are to try to
sell the user fruit.

Below is your context:
Apple
Description: A crisp and juicy fruit with a rounded shape...

Banana
Description: This is a long, curved fruit with a soft,
creamy white flesh...

Orange
Description: A round citrus fruit known for its tough,
bright orange skin...

Strawberry
Description: A small, heart-shaped fruit that is bright
red when ripe...

Watermelon
Description: A very large, heavy fruit with a thick,
striped green rind...

Rules:
1. Always stay on topic.
2. Avoid unnecessary information.
3. Provide clear and direct responses only.
4. Reply in no more than a sentence.
'''

Typical output looked like this:

Try a crisp apple for a versatile snack, a sweet banana
for energy on the go, or a juicy orange for a refreshing
vitamin C boost!

Every response followed the same pattern: apples first, sometimes bananas, always oranges. Never strawberries. Never watermelons.

The knee-jerk fix (and why it didn’t work)

Without root cause analysis, the obvious fix is to add an instruction:

“All fruits should be represented equally in sales, including Apples, Oranges, Bananas, Strawberries and Watermelon.”

After this change:

Apple: 10, Banana: 3, Orange: 10, Strawberry: 3, Watermelon: 4

At this point, it would be tempting to ship the fix and move on. The numbers improved. But this hasn’t uncovered why the bias exists, which means it’ll resurface in different forms as the product catalogue changes.

Applying root cause analysis

Observability

We decomposed the prompt into its structural parts:

• Instruction: “You are a helpful assistant… try to sell the user fruit.”
• Context: Five product descriptions, ordered Apple → Banana → Orange → Strawberry → Watermelon
• Rules: Stay on topic, avoid unnecessary info, clear responses only, reply in no more than a sentence

Categorisation

The failure counts gave us clear labels: Apple (10), Orange (10), Banana (7), Strawberry (0), Watermelon (0). A clean split between “always mentioned” and “never mentioned.”

Pattern identification

The cluster was obvious: the LLM heavily favoured fruits listed earlier in the context. The ordering in the prompt, Apple, Banana, Orange, Strawberry, Watermelon, correlated almost exactly with mention frequency.

Investigation

Two structural elements in the prompt could explain this:

1. Context ordering, fruits listed earlier get more attention
2. Output length restriction, “reply in no more than a sentence” forces the model to pick a subset, and it picks from the top

Hypothesis testing

We built a counting mechanism and increased test iterations to 50 for statistical significance:

fruit_count = {
    "apple": 0, "banana": 0, "orange": 0,
    "strawberry": 0, "watermelon": 0,
}

for response in list_of_responses:
    response_lower = response.lower()
    for fruit_name, root in fruit_search_roots.items():
        if root in response_lower:
            fruit_count[fruit_name] += 1

Then tested each hypothesis in isolation.

Hypothesis 1: Context order bias

Test: Reverse the context order to Watermelon → Strawberry → Orange → Banana → Apple.

Hypothesis 2: Output length restriction

Test: Remove the rule “reply in no more than a sentence.” Keep original context order.

Apple: 50, Banana: 50, Orange: 49, Strawberry: 46, Watermelon: 49

With the length restriction removed, all five fruits were recommended roughly equally. The model wanted to mention all products, the sentence limit was forcing it to truncate, and truncation favoured items listed first.

Hypotheses 3 & 4: Isolating apple bias vs. positional dominance

The residual 24 apples from Hypothesis 1 raised a question: does the model have an inherent preference for apples (perhaps from training data), or is position always the dominant factor?

Hypothesis 3: Ask the model to pick a random fruit from the list Apple, Banana, Orange, Strawberry, Watermelon.

Apple: 0, Banana: 5, Orange: 12, Strawberry: 20, Watermelon: 13

Zero apples, despite being listed first. When explicitly asked to be random, the model avoided the first item. Interesting, but contradictory to the earlier results.

Hypothesis 4: Same test, reversed list, Watermelon, Strawberry, Orange, Banana, Apple.

Apple: 31, Banana: 3, Orange: 15, Strawberry: 1, Watermelon: 0

Root cause report

The knee-jerk fix was a fragile patch that only masked the symptom. Methodical RCA uncovered the real problem, enabling a permanent, robust fix.

Observed behaviour

The LLM consistently recommended apples, oranges, and bananas while ignoring strawberries and watermelons entirely.

Root causes identified

Two interacting biases:

1. Context order bias, the model strongly favours items listed earlier in the context when generating recommendations
2. Output restriction amplification, the “reply in no more than a sentence” rule forces truncation, which reinforces the order bias by cutting off items the model would otherwise have included

Permanent fixes

Understanding the root cause opens up robust solutions rather than prompt patches:

• Randomise context order on each API call, eliminates positional bias entirely
• Rework the length constraint, use “mention at least one less common option” rather than forcing a single sentence
• Rotate featured products, structurally ensure catalogue coverage over time