liveBook · Manning

Disagreement

As explained in LLM Customization and Fine-Tuning

The proportion of cases where model outputs diverge across runs, prompts, or comparison conditions, used here as a brittleness indicator.

Disagreement rate is the fraction of test cases where repeated runs do not all produce the same output. It complements accuracy: a prompt can score well on average yet vary enough to surprise you in production. section 4.4.3

Use the metric as part of prompt validation:

  • Repeat the same prompt on the same held-out test set, and report disagreement alongside accuracy. section 4.4.3
  • Higher disagreement signals greater brittleness; investigate it alongside other signals rather than treating it as a complete stability verdict. section 4.4.3 section 4.4.5
  • Measure before deployment and again after model upgrades; high disagreement at low temperature may call for simplifying the task or fine-tuning. section 4.4.5

Few-shot ticket classifier

On an 18-case smoke test, an eight-shot prompt produced 0% disagreement across three runs at temperature 0.7. Reducing it to two shots raised disagreement to 5.56%: one case changed across runs. section 4.4.3

Listing 4.3 Prompt validator (prompt_validator.py)
# code/chapter04/prompt_validator.py (key sections -- full source in code/chapter04/prompt_validator.py)
 
from collections import Counter
from dataclasses import dataclass
 
@dataclass
class ValidationReport:
    accuracy_mean: float
    accuracy_runs: list[float]
    disagreement_rate: float
    per_example_consistency: list[float]
 
def validate_prompt(prompt_template, test_set, runs: int = 5,
                    temperature: float = 0.7) -> ValidationReport:
    """Run each test case `runs` times, return accuracy and disagreement."""
    all_outputs = [[] for _ in test_set]
    accuracies = []
    for _ in range(runs):
        run_correct = 0
        for i, case in enumerate(test_set):
            prompt = prompt_template.format(input=case["input"])
            output = call_model(prompt, temperature=temperature)
            all_outputs[i].append(output)
            if output.strip() == case["expected"].strip():
                run_correct += 1
        accuracies.append(run_correct / len(test_set))
 
    consistencies = []
    disagreements = 0
    for outputs in all_outputs:
        most_common_count = Counter(outputs).most_common(1)[0][1]
        consistencies.append(most_common_count / runs)
        if most_common_count < runs:
            disagreements += 1
 
    return ValidationReport(
        accuracy_mean=sum(accuracies) / len(accuracies),
        accuracy_runs=accuracies,
        disagreement_rate=disagreements / len(test_set),
        per_example_consistency=consistencies,
    )

A Python validator reports mean accuracy, accuracy by run, disagreement rate, and per-example consistency. section 4.4.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines disagreement across repeated runs and interprets rates, while treating it as one brittleness signal rather than a complete verdict.
Examples
Compares an eight-shot and a two-shot ticket classifier across runs at temperature 0.7.
What it assumes
Requires a labeled held-out test set, which validation guidance assembles and freezes before prompt tuning.
What you can do after
Evaluate prompt stability by repeating test cases and comparing disagreement alongside accuracy.
How it connects
Connects prompt validation with accuracy, temperature, few-shot examples, and fine-tuning decisions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.