Experimentation
Experimentation is the planned investigation of a question or hypothesis by conducting trials or interventions and observing their outcomes.
Machine Learning Platform Engineering
A/B testing is a controlled comparison of two or more variants by assigning them to different users or inputs and measuring their outcomes.
- How it is used
- Compares deployed model or prompt variants using production traffic and measured outcomes to guide improvements.
- How deep it goes
- Describes traffic allocation and observable prompt outcomes, but gives no statistical test, sample-size rule, or A/B implementation code.
- Examples
- Contrasts a pipeline-produced model with one using different preprocessing, and allocates traffic across prompt versions.
- What you can do after
- Apply it via Langfuse to route traffic among prompt variants and compare response quality, satisfaction, and business metrics.
Read how Machine Learning Platform Engineering explains experimentation →
Experimentation for Engineers
A controlled investigation that measures the effect of changing one or more system factors while accounting for other influences.
- How it is used
- Measures the production business impact of system modifications for continuous engineering optimization.
- How deep it goes
- Defines causal measurement needs, details A/B design safeguards, and connects adaptive methods without deriving their internals.
- Examples
- Uses a search-server transformation change, bounce-rate measurement, exchange-cost comparisons, and trading simulation.
- What you can do after
- Apply it via A/B tests to decide whether a production version should replace its baseline.
Read how Experimentation for Engineers explains experimentation →
Effective Data Analysis
An experiment is a procedure that evaluates the effect of an intervention by comparing outcomes under different conditions.
- How it is used
- Guides product and marketing decisions by testing how changes affect user behavior and business metrics.
- How deep it goes
- Explains experimental conditions, group assignment, comparisons, and interpretation, while deferring detailed statistical tests and metrics.
- Examples
- Uses website button variations and a shopping-cart abandonment exercise with three layouts and a control.
- What you can do after
- Practice designing and interpreting group comparisons, including hypotheses, outcome measures, stakeholder context, and possible follow-ups.
First-Party Data Activation
Experimentation is the systematic use of experiments to compare marketing interventions and learn which produce better outcomes.
- How it is used
- Uses controlled experiments to identify effective audiences and campaigns as first-party data activation scales.
- How deep it goes
- Explains treatment and control groups, experiment types, measurement choices, and feedback loops without detailed statistical procedures.
- Examples
- Compares personalized and generic email subject lines, and tests percentage discounts against free shipping for dormant customers.
- What you can do after
- Apply controlled comparisons to assess marketing interventions and use results to refine targeting, messaging, and channels.
Read how First-Party Data Activation explains experimentation →
Machine Learning System Design
An A/B test is a controlled experiment that compares outcomes for users or units exposed to two alternative variants.
- How it is used
- Uses controlled online comparisons to assess whether an ML system change improves business-relevant outcomes.
- How deep it goes
- Works through experiment design, simulation, monitoring, and reporting; lists alternatives without detailed treatment.
- Examples
- Illustrates a retail forecasting test using distribution-center groups, average check, and Welch’s t-test.
- What you can do after
- Evaluate and understand an experiment’s design and results, including expected effects, error rates, and rollout implications.
Read how Machine Learning System Design explains experimentation →
The Art of AI Product Development
Experimentation is the deliberate variation and evaluation of product or system conditions to learn what works.
- How it is used
- Guides iterative AI product decisions under uncertainty, from opportunity discovery and prompting to post-launch improvement.
- How deep it goes
- Offers iterative guidelines and a recommendation comparison, without detailing experimental design or statistical analysis.
- Examples
- Compares recommendation strategies using engagement and conversion metrics, and tries alternative prompt formulations before analyzing output errors.
- What you can do after
- Evaluate AI product alternatives using user and business signals, then iterate with feedback and shared team goals.
Read how The Art of AI Product Development explains experimentation →
Platform Engineering on Kubernetes
A/B testing is a technique that compares alternative versions of an application or feature by exposing different user groups to each version and measuring their outcomes.
- How it is used
- Enables teams to compare user-facing alternatives and use outcome data to choose which feature version to keep.
- How deep it goes
- Explains alternative implementations and routing choices, including a Knative traffic configuration, but does not specify measurement or cohort-assignment procedures.
- Examples
- Uses page layouts, registration flows, and tagged frontend revisions as examples of alternatives and routes.
- What you can do after
- Apply it via Knative Serving to expose alternative revisions, while defining user-routing rules and collecting data for a business decision.
Read how Platform Engineering on Kubernetes explains experimentation →
Tech Startup Toolkit
A controlled experiment is a structured test that changes selected conditions while observing outcomes to evaluate a hypothesis.
- How it is used
- Uses controlled tests to validate go-to-market choices when resources are limited and markets are unforgiving.
- How deep it goes
- Defines experiment targets and links testing to business-model verification, but does not detail experimental design or measurement.
- Examples
- Uses market adjacencies, product functionality, channel-partner types, alliance structures, and lighthouse customers as test cases.
- What you can do after
- Apply controlled tests to evaluate go-to-market options and revise the business model.
AI-Powered Search
A/B testing is an experiment that compares two or more variants by randomly assigning users and measuring outcome differences.
- How it is used
- Validates deployed learning-to-rank models by comparing user outcomes across randomly assigned ranking variants.
- How deep it goes
- Defines random assignment, controls, outcomes, and simulation, then stops short of broader experimentation methodology.
- Examples
- Uses two ranking models, the query "transformers dvd," simulated sessions, and purchase counts as the comparison artifact.
- What you can do after
- Apply it via Python simulations to compare ranking models using tracked purchases and identify offline-to-production failures.
Effective Platform Engineering
Experimentation is the systematic evaluation of alternative designs or implementations under realistic conditions to inform decisions.
- How it is used
- Validates platform technology and feature decisions through realistic implementation, measured value, and early user feedback.
- How deep it goes
- Requires proof-of-concept implementation and alpha-user testing, emphasizing incremental evidence rather than abstract technology selection.
- Examples
- Uses actual-platform proofs of concept, Kubernetes service evaluations, and early tests of fundamental features with alpha users.
- What you can do after
- Evaluate platform technologies and feature designs before commitment, using implementation results and user feedback to guide the next decision.
Read how Effective Platform Engineering explains experimentation →
Architecture Modernization: Socio-technical alignment of software, strategy, and structure
A/B testing is an experiment that compares two variants of a product or experience with different users to evaluate their relative outcomes.
- How it is used
- Supports modernization and uncertain discovery by helping teams test technical and organizational approaches before committing.
- How deep it goes
- Explains trust, leadership support, throwaway code, reprioritization risks, and transitions, without detailing experimental measurement or implementation mechanics.
- Examples
- Uses CloudSuite’s workshops, hands-on coding, cross-team API modularization, Mob Programming, and big bet future cores as examples.
- What you can do after
- Apply an experimental mindset by creating space to try approaches, learn from results, and adjust direction as uncertainty decreases.
Think Like a Software Engineering Manager
Experimentation is the deliberate testing of ideas or changes to learn from observed results.
- How it is used
- Uses experimentation to compare market-specific changes and support iterative delivery through feedback and learning.
- How deep it goes
- Explains experimentation through workplace scenarios and iterative cycles, stopping short of implementation detail or statistical analysis.
- Examples
- Uses configurable return limits across marketplaces and partial product delivery followed by feedback.
- What you can do after
- Apply it via iterative delivery to test changes, recognize failures quickly, gather feedback, and learn from results.
Read how Think Like a Software Engineering Manager explains experimentation →
WordPress Power Toolkit: Harness AI to build next-level websites
A/B testing is an experimental technique that compares two or more variants to determine which performs better against a chosen outcome.
- How it is used
- Uses controlled page or campaign variations to improve marketing performance and understand visitor responses.
- How deep it goes
- Defines comparison and shows tool-assisted optimization, but stops short of explaining experimental design or statistical analysis.
- Examples
- Demonstrates testing pop-ups and forms, then pairs results with heatmaps, scroll maps, recordings, and engagement analysis.
- What you can do after
- Apply it via OptinMonster or Crazy Egg to compare variations and improve pages or campaigns.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains experimentation →
Delivering Machine Learning Projects: From design to deployment
An experiment is a structured investigation in which an uncertain proposition is tested through planned observations or actions.
- How it is used
- Guides machine-learning model development from exploratory investigation toward useful, reproducible results.
- How deep it goes
- Explains managed experimentation, upfront records, reproducibility, and data-leakage risks without detailing specific algorithms.
- Examples
- Uses lab-book records, documented model designs, validation-data optimization, and test-data protection as concrete practices.
- What you can do after
- Practice/perform structured model investigations that preserve evidence, reduce repeated work, and guard against misleading evaluation.
Read how Delivering Machine Learning Projects: From design to deployment explains experimentation →
Build High Value Software Systems
A deliberately limited activity undertaken to learn about an uncertain idea or approach.
- How it is used
- Uses short trials to reduce uncertainty before architectural commitment or to test reversible workflow improvements.
- How deep it goes
- Explains spike purpose and evaluation criteria, then compares trade-offs; prototype implementation details are not shown.
- Examples
- Uses notification-system spikes, a staging-deploy script, and a pre-commit formatter trial.
- What you can do after
- Evaluate competing architecture options through spikes, or test local improvements and share measured results.
Read how Build High Value Software Systems explains experimentation →
Introduction to Generative AI, Second Edition
A/B testing is an experimental method that compares alternative versions by measuring their performance under controlled conditions.
- How it is used
- Measures how users respond to competing models or prompts in production.
- How deep it goes
- Explains the comparison setting and outcome metrics but stops short of experimental design or statistical procedures.
- Examples
- Compares two models or prompts using engagement, satisfaction, or other selected production metrics.
- What you can do after
- Evaluate user preferences empirically instead of relying only on a small sample of inspected responses.
Read how Introduction to Generative AI, Second Edition explains experimentation →
Causal AI
An A/B test is an experiment that compares outcomes under two assigned variants, commonly labeled A and B.
- How it is used
- Frames product or policy changes as interventions whose outcome differences should be estimated through random assignment.
- How deep it goes
- Explains assignment, intervention, group means, significance testing, costs, and causal complications, but defers detailed experimental diagnostics.
- Examples
- Uses an online game, side-quest engagement variants, in-game purchases, summary statistics, and a Python significance test.
- What you can do after
- Apply via Python by comparing randomized groups, summarizing outcomes, and testing their mean difference.
Microservices Patterns, Second Edition
The practice of evaluating a technology through controlled trials that approximate the application's real operational needs.
- How it is used
- Evaluates whether a new technology fits an application's needs and the organization's context.
- How deep it goes
- Compares toy, full-production, and single-subdomain trials, then recommends the latter without detailing experiment execution.
- Examples
- Uses Rust adoption to contrast toy applications, a complex production application, and a deployed subdomain.
- What you can do after
- Evaluate technology fit by implementing and deploying one application subdomain with the candidate technology.
Read how Microservices Patterns, Second Edition explains experimentation →
Timeless Algorithms, The Foundational Ideas
An experiment is an information-gathering procedure that can reveal something about uncertain states before a decision.
- How it is used
- Frames information gathering as a decision component: learn before selecting a final act when evidence may alter the preferred choice.
- How deep it goes
- Defines experiments within acts, states, and experiments; explains belief updates and cost tradeoffs, but gives no experiment-design procedure.
- Examples
- Uses diagnostic tests, market surveys, pilot programs, and prototype trials to illustrate information-gathering experiments.
- What you can do after
- Evaluate whether to acquire information by weighing its expected decision value against costs, delays, or risks.
Read how Timeless Algorithms, The Foundational Ideas explains experimentation →
Designing AI Systems
An experiment is a controlled procedure that assigns alternatives to traffic and measures their outcomes.
- How it is used
- Deploys live-traffic comparisons to improve prompts, models, retrieval configurations, and other AI workflow choices.
- How deep it goes
- Defines variants, allocation, metrics, hashing, workflow hooks, and statistical comparison, but does not detail statistical methods.
- Examples
- Uses patient-intake prompt variants, 50/50 traffic, quality scores, turns to resolution, and cost as comparison artifacts.
- What you can do after
- Operate production comparisons that assign stable variants, collect outcomes, and recommend a winner after sufficient samples.
Also covered in
- Causal Inference for Data Science
- Designing Deep Learning Systems: A guide for software engineers
- Data Without Labels
- BDD in Action, Second Edition
- Authorization in Action
- Latency
- The Art of Unit Testing 3e
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- Cloud Observability in Action
- Deep Learning with R, Third Edition
- Deep Learning with Python, Third Edition
- Effective Behavior-Driven DevelopmentIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- biasBias is a systematic tendency or deviation that distorts measurements, estimates, judgments, or outcomes relative to a relevant reference or target.
- meanThe mean is a measure of central location equal to the arithmetic average of numerical observations, or the probability-weighted average of a random variable’s possible values.
- performancePerformance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
Related concepts
- set-based designused by · An engineering approach that explores multiple design options in parallel, narrows them as evidence arrives, and delays commitment until further exploration is no longer worthwhile.
- control groupuses · A control group is a set of experimental units assigned to a baseline or comparison condition for evaluating an intervention or alternative.
- study designincludes · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons for investigating a question and supporting valid conclusions.
- observational datacontrasts with · Observational data are measurements collected without the investigator assigning or controlling the conditions, treatments, or interventions being studied.
- canary deploymentcontrasts with · A canary deployment is a release strategy that exposes a change to a limited portion of users, traffic, or infrastructure before expanding it more broadly.
- structural causal modelmeasures · A structural causal model represents causal relationships among variables through equations that determine each variable from other variables and exogenous factors.
Questions about experimentation
What is experimentation?
Experimentation is the planned investigation of a question or hypothesis by conducting trials or interventions and observing their outcomes.
Which Manning books explain experimentation?
Machine Learning Platform Engineering, Experimentation for Engineers, Effective Data Analysis, First-Party Data Activation, Machine Learning System Design, The Art of AI Product Development, Platform Engineering on Kubernetes, Tech Startup Toolkit and 12 more.
What should I know before learning experimentation?
Manning books that teach experimentation usually introduce language model, bias, mean and performance first.
What is the difference between experimentation and observational data?
Experimentation is the planned investigation of a question or hypothesis by conducting trials or interventions and observing their outcomes. Observational data are measurements collected without the investigator assigning or controlling the conditions, treatments, or interventions being studied.
What kinds of experimentation are there?
Kinds of experimentation include study design.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



















