Disagreement rate is the fraction of test cases where repeated runs do not all produce the same output. It complements accuracy: a prompt can score well on average yet vary enough to surprise you in production. section 4.4.3
Use the metric as part of prompt validation:
- Repeat the same prompt on the same held-out test set, and report disagreement alongside accuracy. section 4.4.3
- Higher disagreement signals greater brittleness; investigate it alongside other signals rather than treating it as a complete stability verdict. section 4.4.3 section 4.4.5
- Measure before deployment and again after model upgrades; high disagreement at low temperature may call for simplifying the task or fine-tuning. section 4.4.5
Few-shot ticket classifier
On an 18-case smoke test, an eight-shot prompt produced 0% disagreement across three runs at temperature 0.7. Reducing it to two shots raised disagreement to 5.56%: one case changed across runs. section 4.4.3
Listing 4.3 Prompt validator (prompt_validator.py)
# code/chapter04/prompt_validator.py (key sections -- full source in code/chapter04/prompt_validator.py)
from collections import Counter
from dataclasses import dataclass
@dataclass
class ValidationReport:
accuracy_mean: float
accuracy_runs: list[float]
disagreement_rate: float
per_example_consistency: list[float]
def validate_prompt(prompt_template, test_set, runs: int = 5,
temperature: float = 0.7) -> ValidationReport:
"""Run each test case `runs` times, return accuracy and disagreement."""
all_outputs = [[] for _ in test_set]
accuracies = []
for _ in range(runs):
run_correct = 0
for i, case in enumerate(test_set):
prompt = prompt_template.format(input=case["input"])
output = call_model(prompt, temperature=temperature)
all_outputs[i].append(output)
if output.strip() == case["expected"].strip():
run_correct += 1
accuracies.append(run_correct / len(test_set))
consistencies = []
disagreements = 0
for outputs in all_outputs:
most_common_count = Counter(outputs).most_common(1)[0][1]
consistencies.append(most_common_count / runs)
if most_common_count < runs:
disagreements += 1
return ValidationReport(
accuracy_mean=sum(accuracies) / len(accuracies),
accuracy_runs=accuracies,
disagreement_rate=disagreements / len(test_set),
per_example_consistency=consistencies,
)A Python validator reports mean accuracy, accuracy by run, disagreement rate, and per-example consistency. section 4.4.3
