Importance sampling
Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
Reinforcement Learning from Human Feedback
Importance sampling is a method for estimating an expectation under one probability distribution using samples drawn from another distribution and probability weights.
- How it is used
- Corrects distribution mismatches when reusing policy rollouts and training with separate sampler and learner systems.
- How deep it goes
- Derives the expectation identity and policy ratios, then develops clipping, truncation, objectives, and an implementation.
- Examples
- Uses reused rollout batches, sampler-versus-learner token probabilities, and a PyTorch per-token loss calculation.
- What you can do after
- Apply it via PyTorch to cap sampler-learner weights and adjust a per-token policy-gradient loss.
Read how Reinforcement Learning from Human Feedback explains importance sampling →
Machine Learning Algorithms in Depth
Importance sampling is a Monte Carlo technique that estimates expectations or integrals under a target distribution by weighting samples drawn from a proposal distribution.
- How it is used
- Estimates target-distribution expectations and integrals by sampling an easier proposal distribution.
- How deep it goes
- Derives weighted Monte Carlo estimation and implements sampling, filtering, density evaluation, and averaging, while deferring further normalization details.
- Examples
- Uses a sinusoidal integrand, an unnormalized chi target, a scaled Gaussian proposal, positive filtering, and numerical ground truth.
- What you can do after
- Build it from scratch by coding a proposal sampler, importance weights, filtered samples, and a weighted Monte Carlo estimate.
Read how Machine Learning Algorithms in Depth explains importance sampling →
Causal AI
Importance sampling estimates expectations under a target distribution using samples from another distribution and likelihood-based weights.
- How it is used
- Supports probabilistic inference when conditioning would discard many generated samples.
- How deep it goes
- Demonstrates weighted Monte Carlo inference and a Pyro workflow, but defers broader inference algorithms and causal applications.
- Examples
- Uses a Pyro transportation model to estimate education probabilities for train riders.
- What you can do after
- Apply it via Pyro to condition a model, sample an empirical marginal, and estimate target probabilities.
What to know first
- conditional probabilityConditional probability is the probability of an event or variable value given that specified information, an event, or another variable’s value is known.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- importance weightproduces · An importance weight is a probability ratio that reweights an outcome or sample to account for a difference between the distribution that generated it and a target distribution.
- proposal distributionuses · A proposal distribution is a probability distribution from which a sampling or inference algorithm generates candidate states or samples.
- Group Relative Policy Optimizationused by · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- monte carlo integrationuses · Monte Carlo integration is a numerical integration technique that estimates an integral using random samples.
- probabilistic reasoningkind of · Probabilistic reasoning is drawing conclusions or making decisions under uncertainty by representing and evaluating possibilities using probabilities.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
Questions about importance sampling
What is importance sampling?
Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
Which Manning books explain importance sampling?
Reinforcement Learning from Human Feedback, Machine Learning Algorithms in Depth and Causal AI.
What should I know before learning importance sampling?
Manning books that teach importance sampling usually introduce conditional probability and parameter first.
What is importance sampling a kind of?
Importance sampling is a kind of probabilistic reasoning. Probabilistic reasoning is drawing conclusions or making decisions under uncertainty by representing and evaluating possibilities using probabilities.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


