liveBook · Manning

Run-to-run noise

As explained in LLM Customization and Fine-Tuning

A measure of how often repeated model runs produce different predictions for the same input.

A ticket classifier can appear accurate in one test run yet assign a different category to the same ticket on another. Run-to-run noise measures this instability: the disagreement rate is the fraction of test cases for which repeated runs do not all produce the same prediction. section 4.4.3

To measure prompt stability:

  • Repeat the same prompt on the same held-out test set, keeping the test conditions consistent. section 4.4.3
  • Count cases with at least one differing prediction across runs, then divide by the number of test cases. Report this rate alongside accuracy. section 4.4.3
  • Use temperature 0.7 to expose brittleness that greedy decoding may hide; a production classifier would typically run at temperature 0. Even at zero, batching, hardware, and library versions can introduce small numerical perturbations. section 4.4.3

Ticket-classifier stability check

On 18 held-out tickets, an eight-shot Qwen3-4B-Instruct-2507 prompt produced 61.1% mean accuracy and 0% disagreement across three runs at temperature 0.7. A two-shot prompt disagreed on one ticket, or 5.56%. The small set makes accuracy imprecise: one ticket accounts for 5.6 percentage points. Zero disagreement means the runs agreed, not that their answers were correct. section 4.4.3

At temperature 0.7, a 5% disagreement rate is an example of a stable prompt; 30% indicates brittleness. These figures accompany, rather than replace, accuracy. Run-to-run variability is also distinct from sampling error: a roughly 20-case set can expose a broken prompt but cannot reliably rank close variants. Use around 50 cases to resolve large gaps and a few hundred to report small differences. section 4.4.3 section 4.4.2

Disagreement can serve as a proxy when token-level log probabilities are unavailable for a confidence-based follow-up workflow. For example, unstable ticket predictions can signal a need to ask the user to disambiguate. Tune workflow thresholds on held-out data rather than treating them as universal defaults. section 4.2.6

Listing 4.3 Prompt validator (prompt_validator.py)
# code/chapter04/prompt_validator.py (key sections -- full source in code/chapter04/prompt_validator.py)
 
from collections import Counter
from dataclasses import dataclass
 
@dataclass
class ValidationReport:
    accuracy_mean: float
    accuracy_runs: list[float]
    disagreement_rate: float
    per_example_consistency: list[float]
 
def validate_prompt(prompt_template, test_set, runs: int = 5,
                    temperature: float = 0.7) -> ValidationReport:
    """Run each test case `runs` times, return accuracy and disagreement."""
    all_outputs = [[] for _ in test_set]
    accuracies = []
    for _ in range(runs):
        run_correct = 0
        for i, case in enumerate(test_set):
            prompt = prompt_template.format(input=case["input"])
            output = call_model(prompt, temperature=temperature)
            all_outputs[i].append(output)
            if output.strip() == case["expected"].strip():
                run_correct += 1
        accuracies.append(run_correct / len(test_set))
 
    consistencies = []
    disagreements = 0
    for outputs in all_outputs:
        most_common_count = Counter(outputs).most_common(1)[0][1]
        consistencies.append(most_common_count / runs)
        if most_common_count < runs:
            disagreements += 1
 
    return ValidationReport(
        accuracy_mean=sum(accuracies) / len(accuracies),
        accuracy_runs=accuracies,
        disagreement_rate=disagreements / len(test_set),
        per_example_consistency=consistencies,
    )

The prompt validator organizes repeated runs into accuracy and disagreement reporting. section 4.4.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the rate, specifies a repeated-test procedure, and interprets results alongside accuracy and small-sample uncertainty.
Examples
Compares eight-shot and two-shot ticket classification across repeated runs on 18 held-out cases.
What you can do after
Evaluate/understand prompt stability by calculating disagreement alongside accuracy and distinguishing it from sampling error.
How it connects
Connects disagreement to held-out evaluation, decoding temperature, confidence calibration, and few-shot prompting.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.