A ticket classifier can appear accurate in one test run yet assign a different category to the same ticket on another. Run-to-run noise measures this instability: the disagreement rate is the fraction of test cases for which repeated runs do not all produce the same prediction. section 4.4.3
To measure prompt stability:
- Repeat the same prompt on the same held-out test set, keeping the test conditions consistent. section 4.4.3
- Count cases with at least one differing prediction across runs, then divide by the number of test cases. Report this rate alongside accuracy. section 4.4.3
- Use temperature 0.7 to expose brittleness that greedy decoding may hide; a production classifier would typically run at temperature 0. Even at zero, batching, hardware, and library versions can introduce small numerical perturbations. section 4.4.3
Ticket-classifier stability check
On 18 held-out tickets, an eight-shot Qwen3-4B-Instruct-2507 prompt produced 61.1% mean accuracy and 0% disagreement across three runs at temperature 0.7. A two-shot prompt disagreed on one ticket, or 5.56%. The small set makes accuracy imprecise: one ticket accounts for 5.6 percentage points. Zero disagreement means the runs agreed, not that their answers were correct. section 4.4.3
At temperature 0.7, a 5% disagreement rate is an example of a stable prompt; 30% indicates brittleness. These figures accompany, rather than replace, accuracy. Run-to-run variability is also distinct from sampling error: a roughly 20-case set can expose a broken prompt but cannot reliably rank close variants. Use around 50 cases to resolve large gaps and a few hundred to report small differences. section 4.4.3 section 4.4.2
Disagreement can serve as a proxy when token-level log probabilities are unavailable for a confidence-based follow-up workflow. For example, unstable ticket predictions can signal a need to ask the user to disambiguate. Tune workflow thresholds on held-out data rather than treating them as universal defaults. section 4.2.6
Listing 4.3 Prompt validator (prompt_validator.py)
# code/chapter04/prompt_validator.py (key sections -- full source in code/chapter04/prompt_validator.py)
from collections import Counter
from dataclasses import dataclass
@dataclass
class ValidationReport:
accuracy_mean: float
accuracy_runs: list[float]
disagreement_rate: float
per_example_consistency: list[float]
def validate_prompt(prompt_template, test_set, runs: int = 5,
temperature: float = 0.7) -> ValidationReport:
"""Run each test case `runs` times, return accuracy and disagreement."""
all_outputs = [[] for _ in test_set]
accuracies = []
for _ in range(runs):
run_correct = 0
for i, case in enumerate(test_set):
prompt = prompt_template.format(input=case["input"])
output = call_model(prompt, temperature=temperature)
all_outputs[i].append(output)
if output.strip() == case["expected"].strip():
run_correct += 1
accuracies.append(run_correct / len(test_set))
consistencies = []
disagreements = 0
for outputs in all_outputs:
most_common_count = Counter(outputs).most_common(1)[0][1]
consistencies.append(most_common_count / runs)
if most_common_count < runs:
disagreements += 1
return ValidationReport(
accuracy_mean=sum(accuracies) / len(accuracies),
accuracy_runs=accuracies,
disagreement_rate=disagreements / len(test_set),
per_example_consistency=consistencies,
)The prompt validator organizes repeated runs into accuracy and disagreement reporting. section 4.4.3
