Importance sampling
Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
Reinforcement Learning from Human Feedback
Importance sampling is a method for estimating an expectation under one probability distribution using samples drawn from another distribution and probability weights.
- How it is used
- Corrects distribution mismatch so policy gradients can reuse rollouts from older policies or separate sampling systems.
- How deep it goes
- Derives the expectation identity and policy ratios, then compares clipping, sequence-level ratios, and truncated weights with implementation details.
- Examples
- Uses old-policy rollout reuse, rare reasoning tokens, sequence-level GSPO ratios, and learner-versus-sampler TIS code.
- What you can do after
- Apply it via PPO, GSPO, or TIS ratios to correct rollout and sampler-learner distribution mismatches.
Read how Reinforcement Learning from Human Feedback explains importance sampling →
Machine Learning Algorithms in Depth
Importance sampling is a Monte Carlo technique that estimates expectations or integrals under a target distribution by weighting samples drawn from a proposal distribution.
- How it is used
- Estimates target-distribution expectations and integrals by sampling an easier proposal distribution.
- How deep it goes
- Derives weighted Monte Carlo estimation and implements sampling, filtering, density evaluation, and averaging, while deferring further normalization details.
- Examples
- Uses a sinusoidal integrand, an unnormalized chi target, a scaled Gaussian proposal, positive filtering, and numerical ground truth.
- What you can do after
- Build it from scratch by coding a proposal sampler, importance weights, filtered samples, and a weighted Monte Carlo estimate.
Read how Machine Learning Algorithms in Depth explains importance sampling →
Causal AI
Importance sampling estimates expectations under a target distribution using samples from another distribution and likelihood-based weights.
- How it is used
- Supports probabilistic inference when conditioning would discard many generated samples.
- How deep it goes
- Demonstrates weighted Monte Carlo inference and a Pyro workflow, but defers broader inference algorithms and causal applications.
- Examples
- Uses a Pyro transportation model to estimate education probabilities for train riders.
- What you can do after
- Apply it via Pyro to condition a model, sample an empirical marginal, and estimate target probabilities.
Related concepts
- importance weightproduces · An importance weight is the ratio of an outcome’s probability under a target distribution or policy to its probability under a sampling or reference distribution or policy, used to reweight that outcome.
- proposal distributionuses · A proposal distribution is a probability distribution from which a sampling or inference algorithm generates candidate states or samples.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- Group Relative Policy Optimizationused by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
- Pyroused by · Pyro is an open-source probabilistic programming library for building probabilistic models and performing Bayesian inference with Python and PyTorch.
- monte carlo integrationuses · Monte Carlo integration is a numerical integration technique that estimates an integral using random samples.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.