liveBook · Manning

Sampling

As explained in Agent Design Patterns

Sampling is the process of generating or selecting an output from a stochastic model or operation.

An agent can give different answers to the same input. When a language model samples predicted tokens at a generation temperature above zero, each run may take a different path; a fixed random seed can instead make the process deterministic. This variability can cause failures, but it also gives an agent multiple candidates to inspect. section 6.0

Sampling is the production of a candidate output from a stochastic worker. Repeated sampling gives an agent two distinct options: retry until an output passes a criterion, or collect separate outputs to compare and reconcile. section 5.2 section 6.0

In a resampler, sampling is a bounded retry loop:

  • Invoke the worker on the input and assess its candidate against a prespecified criterion. Return the candidate immediately if it passes. section 5.2
  • If it fails, invoke the worker again with the same input. Do not pass back the rejected candidate or its failure reason; the next attempt depends on the worker's stochasticity, not feedback. section 5.2
  • Cap the number of trials. Cost grows linearly with trials; after the cap, return a placeholder or raise an error according to the criterion's importance. section 5.2
Listing 5.1 resampler.py: Generic resampler
from ..utils import E, IE
from typing import Callable
 
 
class Resampler(IE[[str], str]):
  def __init__(
    self,
    worker: IE[[str], str],
    assessor: E[[str], str],
    branching: Callable[[str], bool],
    n_max_trials: int,
  ):
    self.worker = worker
    self.assessor = assessor
    self.branching = branching
    self.n_max_trials = n_max_trials
 
  async def __call__(self, input: str) -> str:
    n_trials: int = 0
    while n_trials < self.n_max_trials:
      result = await self.worker(input)
      assessment: str = await self.assessor(result)
      criterion_satisfied: bool = self.branching(
        assessment
      )
      if criterion_satisfied:
        return result
      n_trials += 1
    raise ResamplingFailureError(
      f"Failed to produce a valid result in {n_trials} trials"
    )
 
 
class ResamplingFailureError(Exception):
  pass

The generic Python implementation makes the boundary explicit: the worker produces a result, the assessor evaluates it, and a branching function decides whether to return or retry. Exhausting the trial limit raises ResamplingFailureError. section 5.2

A Voter uses repeated runs differently. It treats their answers as a distribution and reconciles them to seek a more accurate prediction or estimate confidence, rather than stopping at the first answer that passes a criterion. section 6.0

Multiple paths through a math problem

Ask the same model repeatedly to count ordered triples of positive integers whose product is 108 and whose values form a non-degenerate triangle. Different runs may follow different reasoning paths; ensemble sampling makes those paths available for comparison. section 6.0

How Agent Design Patterns covers it

How deep it goes
Explains token-level variability and implements a bounded assessment loop, but does not detail the model's sampling algorithm.
Examples
Illustrates sampling with a generic Python resampler and an ordered-triples mathematics problem posed for repeated model runs.
What it assumes
Introduces generation temperature while using worker interfaces and asynchronous Python calls without defining those programming constructs.
What you can do after
Build it from scratch as a bounded resampler with a worker, assessor, branching function, and failure path.
How it connects
Connects sampling to stochasticity, generation temperature, resamplers, ensemble sampling, and Voters.

Also explained in

Compare how 31 books explain sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Agent Design Patterns. The section links point to the parts of the book it draws on.