Chapter 8: Prompt sampling

Introduction

The introduction explains why a strong prompt may still produce different answers across calls and why sampling multiple responses can make decisions more reliable.

• Reliability through prompt sampling: Explain why a strong prompt can produce different answers across independent calls and how collecting and aggregating or selecting multiple generations can improve reliability. Use the runbook scoring example to show why one response may be insufficient for a release decision.
  Why it matters: Sampling matters when variation between calls could change a consequential decision.

Key terms: prompt sampling, independent generations, aggregation, selection method

8.1 Sampling fundamentals

This section explains why a single response may be insufficient and defines prompt sampling as generating and combining multiple responses.

• Prompt sampling rationale and workflow: Explain why one generation may be enough for a low-consequence task but unreliable for a higher-consequence task, and describe how prompt sampling uses multiple independent responses to the same prompt.
  Why it matters: This gives you a reason to consider multiple generations when relying on just one response is risky.

Key terms: single generation, sample, probability distribution, prompt sampling, independent responses, aggregates

8.1.1 Temperature and diversity

Explains how temperature affects the diversity and quality of outputs used in prompt sampling.

• Bounded temperature for sampling: Explain why sampling needs enough temperature to produce varied outputs but not so much that noise undermines aggregation, and describe the suggested starting range.
  Why it matters: The temperature setting affects whether repeated generations provide useful variety or mostly noise.

Key terms: temperature, output variability, aggregation, moderate temperature

8.1.2 Sample size

This subsection gives starting sample sizes for different tasks and explains when to use fewer or more generations.

• Task-Based Sample Size: Describe the suggested number of generations for categorical, convergence, and numerical-smoothing tasks, and explain how input ambiguity affects the choice within those ranges.
  Why it matters: Choosing a sample size that fits the task gives you a practical starting point without generating more outputs than needed.

Key terms: generations, categorical, convergence tasks, numerical smoothing, lower bound, upper bound

8.1.3 Sampling in practice

Shows how to sample responses without code and choose a technique based on the kind of output needed.

• Independent no-code sampling: Describe how to sample without code by submitting the same prompt multiple times in separate chats, collecting the outputs, and then applying a suitable technique.
  Why it matters: Separate chats let you try sampling even when you are using a conversational interface rather than code.
• Output-based technique selection: Distinguish which sampling technique the section pairs with numeric outputs, category labels, analytical responses needing stable reasoning, and full responses needing the strongest candidate.
  Why it matters: The kind of answer you need determines how to use the outputs you collected.

Key terms: sampling, independent chats, generations, mean, mode, self-consistency, best-of-N

8.2 Mean

This section explains when averaging multiple prompt outputs is useful and when another sampling method is a better fit.

• Mean for numeric outputs: Explain why averaging multiple numeric generations can be useful when variation might affect a decision, and distinguish this from cases that call for mode or best-of-N.
  Why it matters: Choosing mean for the right kind of output can keep an unusually high or low score from dominating a decision.

Key terms: mean, numeric outputs, outlier, mode, best-of-N

8.2.1 Practical example: Runbook quality scoring

A runbook-scoring example shows how to average independent numerical ratings and use their spread when judging release readiness.

• Runbook mean scoring workflow: Describe how the SRE team uses one fixed scoring prompt to obtain four independent integer ratings, then calculates their arithmetic mean as the operational score.
  Why it matters: A consistent numerical output makes scores from separate runs straightforward to aggregate.
• Mean and spread interpretation: Explain why the team uses the mean rather than one sampled score for a release-readiness decision, and what a wider spread signals. Distinguish this numerical use of mean from categorical choices, for which the section points to mode.
  Why it matters: The average supports a less run-dependent decision, while disagreement between scores still deserves attention.

Key terms: scoring prompt, independent responses, arithmetic mean, operational score, mean, spread, ambiguity signal, mode

8.3 Mode

This section explains how mode selects a categorical label from multiple generated responses and when that choice is useful.

• Mode as label voting: Explain that mode selects the most frequent categorical label from multiple generated responses, such as severity or routing labels.
  Why it matters: It gives you a way to turn varying categorical responses into one selected label.
• Mode selection criteria: Distinguish when mode is useful from cases better suited to mean or best-of-N, and describe what the distribution of votes can indicate.
  Why it matters: Choosing the method that matches the output prevents treating numbers or full responses as label votes.

Key terms: mode, categorical outputs, majority voting, single label, vote distribution, confidence signal, mean, best-of-N

8.3.1 Practical example: Alert severity routing

This example shows how to route an alert by voting across independent severity classifications and how to interpret the vote split.

• Majority vote alert routing: Trace how the platform team uses one label-only prompt to generate four independent P1, P2, or P3 classifications, counts the votes, and routes by the majority label. State that P1 won three of four votes in the example, supporting escalation.
  Why it matters: Voting can change an alert’s routing when a single classification would send it to a lower-priority queue.
• Vote split confidence context: Explain what the vote split adds to the routing decision and what the team can do when there is no clear majority.
  Why it matters: The vote split helps distinguish a clear routing choice from one that needs further attention.
• Labels versus prose convergence: Distinguish mode voting for single-label outputs from the self-consistency check described for full responses.
  Why it matters: The form of the output determines whether counting labels is useful.

Key terms: mode, label-only classification prompt, majority label, vote split, clear majority, human review, single-label outputs, self-consistency

8.4 Self-consistency

This section explains when to use self-consistency to assess whether independently generated full responses reach the same substantive conclusion.

• Self-consistency use and limits: Explain when self-consistency is useful: a task permits different ways to express a full response, but independent generations should reach the same substantive conclusion. Distinguish this from using mode for category labels or mean for single numeric values.
  Why it matters: Choosing the approach that matches the output keeps you from treating varied wording as disagreement or applying self-consistency to a label or number.

Key terms: self-consistency, independent generations, substantive conclusion, mode, mean

8.4.1 Practical example: Incident root-cause draft

This example shows how to compare multiple incident root-cause drafts for agreement while flagging causal claims the notes do not establish.

• Incident draft comparison workflow: Describe how the on-call engineer uses one fixed prompt to generate independent drafts, normalizes them into shared fields, and selects the convergent conclusion while noting differences for review.
  Why it matters: Shared fields make agreement and differences easier to see across differently worded drafts.
• Convergence and causal uncertainty: Distinguish what the three drafts agree on from where their causal framing differs, and explain why that difference needs review against the incident notes.
  Why it matters: Agreement on the timeline and rollback should not conceal an unsupported explanation of why the incident happened.

Key terms: self-consistency, independent drafts, shared comparison fields, trigger, impact, corrective action, causal framing, divergence, incident notes

8.4.2 Checking self-consistency

This section shows how to compare independent responses for self-consistency, assess their agreement, and decide what to do when they diverge.

• Self-consistency comparison process: Describe how to check self-consistency by generating independent responses, reducing each to a short structured comparison, and identifying their shared conclusion and divergences in a separate pass.
  Why it matters: A structured comparison makes it easier to tell whether different drafts support the same conclusion.
• Divergence and confidence decisions: Explain how the comparison treats minor wording differences versus material divergence, and what to do when responses do not converge.
  Why it matters: It prevents a superficial agreement score from hiding meaningful differences or forcing a conclusion the drafts do not support.

Key terms: self-consistency, independent responses, structured comparison view, shared conclusion, divergences, confidence, best-of-N, predefined criteria

8.5 Best-of-N

Explains when to use best-of-N sampling, how to select a complete response, and why selection criteria should be set in advance.

• Best-of-N selection and fit: Explain that best-of-N generates N complete responses and selects the strongest against a rubric when one final response is needed and quality differences justify comparison. Distinguish this from using mode for labels or mean for numeric scores.
  Why it matters: It helps you choose a complete response without applying a voting or averaging method to the wrong kind of output.
• Criteria before candidate review: Explain why selection criteria should be defined before reading best-of-N candidates, and identify first-response anchoring as the risk of reading them first.
  Why it matters: A rubric set in advance makes candidate selection less vulnerable to first impressions.

Key terms: best-of-N, full candidates, rubric, mode, mean, criteria, anchor

8.5.1 Practical example: Deployment status update for executives

This section demonstrates how to use best-of-N sampling to choose an executive deployment update against criteria set in advance.

• Predefined candidate selection workflow: Describe how the release manager generates and selects one executive-ready update after a failed deployment: set criteria first, generate complete candidates under the same requirements, score them, and archive the rubric.
  Why it matters: A consistent prompt and advance criteria make the final choice explainable rather than arbitrary.
• Executive update selection rationale: Justify why candidate 2 is selected over candidates 1 and 3 using the section’s criteria for an executive status update.
  Why it matters: The strongest draft is the one that gives leadership an actionable account, not merely an accurate description of the incident.

Key terms: best-of-N, candidates, criteria, rubric, auditability, clarity, leadership tone, concrete mitigation, update commitment, release safety framing

8.6 When to use sampling

This section explains when sampling is worth its added generation cost and why sampled outputs still need human review before operational use.

• Sampling decision criteria: Explain when sampling is usually warranted, when it is not, and how the cost of a wrong output affects the decision.
  Why it matters: This check helps you spend extra generations where they can change a consequential outcome.
• Human review of sampled results: Explain why aggregated or selected sampling results still need qualified human review before operational decisions.
  Why it matters: A stronger signal from multiple outputs is not, by itself, approval to act on the result.
• Sampling cost and prompt brevity: Describe the cost tradeoff of additional generations and why prompts should stay concise when sampling.
  Why it matters: Concise prompts keep the cost of repeated generations focused on useful signal.

Key terms: sampling, high consequence classification or routing, numerical scoring, open-ended outputs, generation cost, signal reliability, human review, aggregated scores, majority labels, selected candidates

8.6.1 Hands-on practice

This hands-on section asks readers to apply prompt-sampling methods to software-engineering tasks and justify which method fits each type of output.

• Categorical and numeric aggregation: Apply mode to four independent P1/P2/P3 alert-severity classifications and mean to four numeric runbook scores to produce one release-readiness score.
  Why it matters: The kind of answer a prompt produces affects how you combine multiple generations.
• Draft convergence and update selection: Describe how the exercises use self-consistency to find convergent conclusions across root-cause drafts and best-of-N to select an incident status update using predefined criteria.
  Why it matters: These exercises give different purposes to generating multiple open-ended responses: finding agreement or choosing a complete response.
• Task-specific sampling choices: Justify a sampling-method choice for three internal tasks: one numeric, one categorical, and one open-ended.
  Why it matters: Choosing a method for a real workflow requires explaining why it fits that task.

Key terms: mode, mean, severity-classification prompt, numeric scoring prompt, self-consistency, best-of-N, convergent conclusions, predefined criteria, sampling method, numeric

8.7 Summary

The section summarizes how prompt sampling improves reliability, how to choose a sampling technique, and when uncertain results call for more sampling or escalation.