What you will learn in Experimentation for Engineers
From A/B testing to Bayesian optimization
Chapter by chapter
Chapter 3: Multi-armed bandits: Maximizing business metrics while experimenting
What you learn in chapter 3 →Chapter 4: Response surface methodology: Optimizing continuous parameters
What you learn in chapter 4 →Chapter 6: Bayesian optimization: Automating experimental optimization
What you learn in chapter 6 →Concepts explained in this book
- response surface methodologyA statistical optimization methodology that models a response as a function of controllable factors to identify promising settings.
- study designThe planning of an experiment's configurations, measurements, allocation, and duration to obtain useful results efficiently.
- metricA quantitative measure of business performance, such as revenue, engagement, or time spent using an application.
- thresholdA configurable cutoff value in the trading strategy that determines how the strategy responds to an input or condition.
- surrogate modelA surrogate function is a model that approximates a measured response as a function of parameter values and estimates the response at unmeasured values.
- epsilon-greedyEpsilon-greedy is a bandit allocation method that usually selects the currently best alternative while exploring alternatives at a specified probability.
- interpolationA technique for estimating values between observed values using the relationship among nearby known points.
- markout profitA trading-profit metric measuring the profit associated with a trade after observing subsequent market movement over a specified markout period.
- aggregate metricAn aggregate measurement is a summary statistic combining individual measurements to estimate an overall effect or quantity.
- continuous parameterA system parameter whose value can vary over a continuous numerical range rather than being restricted to discrete choices.
- linear regressionA regression method that models a response as a linear function of explanatory variables.
- Thompson samplingThompson sampling is a Bayesian bandit algorithm that samples each alternative's expected reward from its posterior distribution and selects an alternative according to those samples.
- domain knowledgeDomain knowledge is specialized knowledge about the application area, processes, and constraints relevant to a system.
- standard errorA measure of the sampling variability of an estimated aggregate or statistic.
- experimentationA controlled investigation that measures the effect of changing one or more system factors while accounting for other influences.
- click-through rateClick-through rate is the proportion of opportunities in which users click a presented item, such as an advertisement or article.
- CPU timeCPU time is the amount of processor execution time consumed by a program or process.
- parabolaA quadratic curve whose graph can represent a response changing with one parameter and having an interior maximum or minimum.
- contextual banditA sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.
- multi-armed banditA multi-armed bandit is a sequential decision problem of choosing among alternatives with unknown reward distributions while learning which maximizes cumulative reward.
- false positiveA false positive is an incorrect conclusion that an observed effect or condition is present when it is not.
- randomizationRandomization is the assignment of observations or users to experimental versions by chance so that unrelated factors do not systematically differ between versions.
- individual measurementA single observed value of a metric from one experimental observation.
- replicationReplication is the repeated measurement of the same experimental conditions to reduce the effect of natural variation on an estimate.
- regretRegret is the lost outcome relative to the outcome that would have been obtained by choosing the best alternative throughout the decision process.
- transaction costExecution cost is the expense incurred when carrying out a trade, including the difference between the intended and obtained transaction price and related charges.
- practical importanceThe smallest effect size considered large enough to justify changing a system.
- bootstrap Thompson samplingBootstrap Thompson sampling is a form of Thompson sampling that estimates each alternative's probability of being best from bootstrap resamples or an ensemble of fitted models.
- randomized probability matchingRandomized probability matching is a decision rule that selects each alternative with probability equal to the estimated probability that the alternative is optimal.
- Bayesian optimizationBayesian optimization is a sequential optimization method that uses a probabilistic model and Bayesian updating to choose promising evaluations of an expensive objective.
- acquisition functionAn acquisition function is a criterion that scores candidate evaluations according to their expected value for selecting the next experiment in Bayesian optimization.
- click modelA click model is a predictive model that estimates the likelihood that a user will click a particular advertisement.
- standard deviationA measure of the typical magnitude of deviations from a quantity's mean.
- confidence intervalAn interval constructed from sample data that expresses a specified level of uncertainty about a population quantity or distributional value.
- validation measurementA measurement performed at an estimated optimal setting to check whether the predicted system outcome agrees with observed performance.
- false-positive rateFalse-positive rate is the probability that a statistical test produces a false positive.
- squared exponential kernelThe squared exponential kernel is a Gaussian-process covariance function whose similarity decreases exponentially with squared input distance.
- biasA systematic measurement error that consistently shifts results above or below the value being estimated.
- bootstrap samplingA bootstrap sample is a dataset formed by repeatedly drawing observations with replacement from an observed dataset.
- meanA mean is the arithmetic average of a collection of numerical values.
- ensembleA model ensemble is a collection of predictive models whose predictions or decisions are combined, compared, or otherwise used collectively.
- counterfactual outcomeAn outcome or associated information that would occur under a different system state or intervention than the one observed.
- viewing timeThe elapsed time between a user seeing a displayed post and closing the application, used as a measure of engagement.
- metaparameterA metaparameter is a configuration value that controls the behavior or stopping policy of an adaptive algorithm rather than being learned directly from its observations.
- reward signalA reward is the observed outcome assigned to a contextual-bandit action, represented here by the user's viewing time.
- simulationA computational representation of a system that estimates its business metrics offline by combining data, models, scientific models, or heuristics.
- false negativeA false negative is an error in which a test fails to identify that a better system version should be adopted.
- false-negative rateFalse-negative rate is the probability that a statistical test produces a false negative.
- power analysisPower analysis is the calculation of the sample size or sensitivity needed to limit the probability of failing to detect a specified effect.
- hyperparameterA hyperparameter is a setting that controls the behavior or structure of a statistical model rather than representing an observed system input.
- kernel functionA kernel function assigns a similarity or weight to an observation based on its distance from another input.
- residual errorA residual is the portion of an observed value that remains unexplained after subtracting a model estimate.
- cluster downweightingCluster downweighting reduces the influence of an observation when other observations occur nearby in the input space.
- weighted averageA weighted average is an average in which each value contributes in proportion to an assigned weight.
- exploitationA decision strategy that selects the alternative currently believed to produce the best observed outcome.
- online measurementMeasurement of a change's impact while the change is deployed in a live production system.
- daily active usersDaily active users is the number of distinct users who engage with a product during a given day.
- precisionPrecision is the degree to which repeated estimates or measurements concentrate closely around their expected value.
- armAn arm is an alternative action or candidate system version in a multi-armed bandit whose outcomes can be observed and rewarded.
- cross termA regression term formed by multiplying two distinct predictor variables to represent their joint effect.
- anomalyAn observed data point that lies unusually far from the other observations in a dataset or fitted relationship.
- quadratic modelA response model containing constant, linear, squared, and possibly pairwise-product terms of its predictors.
- quadratic termA model term involving the square or pairwise product of predictor variables to represent curvature or interaction.
- two-dimensional experimentA controlled experiment that evaluates a response across settings of two controllable parameters.
- confounderConfounder bias is bias caused when a condition affecting the outcome is applied differently and consistently to the system variants being compared.
- complexityThe property of a system or procedure arising from many interacting components and making its behavior difficult to predict or execute reliably.
- online bootstrap samplingOnline bootstrap sampling is an incremental bootstrap technique that updates bootstrap estimates as new observations arrive without repeatedly processing the full dataset.
- probability of best armThe probability of best arm is the estimated probability that an arm produces the highest aggregate metric among the alternatives.
- stopping criterionA stopping rule is a criterion that determines when an experiment or sequential decision process should terminate.
- interaction effectAn interaction effect is a joint effect of multiple changes that differs from the sum of their separate effects.
- actionA candidate choice made by a contextual bandit, such as displaying one particular post.
- contextThe observed information about a user that conditions which action a contextual bandit selects.
- pseudo-inverseA generalized matrix inverse that remains defined for some matrices whose ordinary inverse is undefined and agrees with the ordinary inverse when that inverse exists.
- independenceIndependence is the probabilistic property that the outcome of one random event does not affect the outcome of another.
- independent and identically distributedThe iid assumption states that measurements are independent of one another and drawn from the same probability distribution.
- bandit feedbackBandit feedback is observational feedback containing rewards for policy-selected actions while omitting outcomes for unsampled actions.
- just-in-time compilationA just-in-time compiler translates code into machine code during program execution rather than exclusively before execution.
- p-hackingThe misuse of statistical analysis or stopping rules to make results appear statistically significant when the apparent effect is not real.
- Central Limit TheoremA theorem stating that suitably normalized sums or averages of many observations tend toward a normal distribution.
- normal distributionA continuous probability distribution characterized by a symmetric bell-shaped curve determined by its mean and standard deviation.
- probability distributionA mathematical description of the possible values of a random quantity and their probabilities.
- epsilon-greedy decay algorithmA variant of the epsilon-greedy algorithm whose exploration probability decreases according to the number of measurements.
- productionAn operational system serving real users or business processes rather than an offline or test environment.
- adverse selectionA trading condition in which execution is more likely when subsequent market movement makes the trade unfavorable.
- adverse selection costThe profit lost because trades are executed under adverse selection conditions.
- face-centered central composite designA central composite experimental design whose axial points lie at the centers of the faces of the factor-space region.
- identically distributed measurementsMeasurements are identically distributed when they can be modeled as draws from the same probability distribution.
- serial dependenceSerial dependence is correlation between measurements arising from their order or from persistent system dynamics.
- time scaleThe duration required to obtain an individual measurement of a metric or to observe a relevant outcome.
- experimental optimizationAn optimization approach that selects system changes or configurations using costly measurements from experiments on live production systems.
- cross entropy lossA loss metric that measures the discrepancy between a probability distribution predicted by a model and the observed outcome distribution.
- selection biasSystematic measurement error caused by collecting samples under conditions that differ across the populations or variants being compared.
- black boxA function or system whose internal process is not known while its inputs and outputs are observable.
- lower confidence boundA lower confidence bound is an acquisition function that combines a predicted response with a multiple of predictive uncertainty to favor promising or informative evaluations.
- unknown unknownFactors affecting system performance that have not been considered because they are absent from the current model or knowledge.
- common workflowThe iterative process of implementing, evaluating, and measuring proposed changes to an engineered system.
- continuous improvementA feedback loop is a cycle in which a system’s outputs influence subsequent inputs, behavior, or observations.
- approximation errorThe difference between a model's estimated response and the measured response it approximates.
- normal equationAn equation system whose solution gives the least-squares coefficients of a regression model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.