Experimentation
Experimentation is the deliberate testing of an idea, intervention, or alternative by observing outcomes to gather evidence or compare effects.
Machine Learning Platform Engineering
A/B testing is a controlled comparison of two or more variants by assigning them to different users or inputs and measuring their outcomes.
- How it is used
- Compares deployed model or prompt variants using production traffic and measured outcomes to guide improvements.
- How deep it goes
- Describes traffic allocation and observable prompt outcomes, but gives no statistical test, sample-size rule, or A/B implementation code.
- Examples
- Contrasts a pipeline-produced model with one using different preprocessing, and allocates traffic across prompt versions.
- What you can do after
- Apply it via Langfuse to route traffic among prompt variants and compare response quality, satisfaction, and business metrics.
Read how Machine Learning Platform Engineering explains experimentation →
Experimentation for Engineers
A controlled investigation that measures the effect of changing one or more system factors while accounting for other influences.
- How it is used
- Measures the production business impact of system modifications for continuous engineering optimization.
- How deep it goes
- Defines causal measurement needs, details A/B design safeguards, and connects adaptive methods without deriving their internals.
- Examples
- Uses a search-server transformation change, bounce-rate measurement, exchange-cost comparisons, and trading simulation.
- What you can do after
- Apply it via A/B tests to decide whether a production version should replace its baseline.
Read how Experimentation for Engineers explains experimentation →
Effective Data Analysis
An experiment is a procedure that evaluates the effect of an intervention by comparing outcomes under different conditions.
- How it is used
- Guides marketing and product decisions by testing whether changes to experiences affect user behavior and business metrics.
- How deep it goes
- Defines manipulation, measurement, control, and assignment, then outlines A/B testing while deferring detailed statistical tests.
- Examples
- Uses button and website variants, plus a shopping-cart layout exercise with control and experiment groups.
- What you can do after
- Practice designing a controlled comparison, stating hypotheses, choosing outcome metrics, and recommending follow-up from group results.
First-Party Data Activation
Experimentation is the systematic use of experiments to compare marketing interventions and learn which produce better outcomes.
- How it is used
- Identifies which audiences, messages, and channels produce incremental marketing outcomes as campaign activity scales.
- How deep it goes
- Defines treatment and control groups, gives comparison patterns, and addresses measurement setup without detailing statistical calculations.
- Examples
- Compares personalized and generic email, segmented and broad targeting, email with and without SMS, and competing paid-social offers.
- What you can do after
- Practice/perform controlled marketing comparisons and use measured differences to refine targeting, content, and channel choices.
Read how First-Party Data Activation explains experimentation →
Machine Learning System Design
An A/B test is a controlled experiment that compares outcomes for users or units exposed to two alternative variants.
- How it is used
- Uses controlled online comparisons to assess whether an ML change improves goal-linked outcomes rather than merely offline metrics.
- How deep it goes
- Works through hypotheses, metrics, error thresholds, sample sizing, simulation, monitoring, and reporting; names non-A/B alternatives without developing them.
- Examples
- Illustrates a retail forecasting pilot with distribution-center groups, average check, revenue estimates, control metrics, and Welch’s t-test.
- What you can do after
- Practice/perform an A/B test plan, validate its statistical behavior by simulation, monitor its execution, and report its business effect.
Read how Machine Learning System Design explains experimentation →
The Art of AI Product Development
Experimentation is the deliberate variation and evaluation of product or system conditions to learn what works.
- How it is used
- Tests AI product choices against user needs and business outcomes before and after launch.
- How deep it goes
- Offers decision criteria and iterative practices, plus an A/B test scenario, but no statistical test procedure.
- Examples
- Compares two collaborative-filtering strategies using engagement and conversions, and tries alternative one-shot prompts before analyzing output errors.
- What you can do after
- Practice/perform it by testing product or prompt alternatives, reviewing user-centered measures and feedback, and iterating with the team.
Read how The Art of AI Product Development explains experimentation →
Platform Engineering on Kubernetes
A/B testing is a technique that compares alternative versions of an application or feature by exposing different user groups to each version and measuring their outcomes.
- How it is used
- Enables teams to compare user-facing alternatives and use outcome data to choose which feature version to keep.
- How deep it goes
- Explains alternative implementations and routing choices, including a Knative traffic configuration, but does not specify measurement or cohort-assignment procedures.
- Examples
- Uses page layouts, registration flows, and tagged frontend revisions as examples of alternatives and routes.
- What you can do after
- Apply it via Knative Serving to expose alternative revisions, while defining user-routing rules and collecting data for a business decision.
Read how Platform Engineering on Kubernetes explains experimentation →
Tech Startup Toolkit
A controlled experiment is a structured test that changes selected conditions while observing outcomes to evaluate a hypothesis.
- How it is used
- Uses controlled tests to validate go-to-market choices when resources are limited and markets are unforgiving.
- How deep it goes
- Defines experiment targets and links testing to business-model verification, but does not detail experimental design or measurement.
- Examples
- Uses market adjacencies, product functionality, channel-partner types, alliance structures, and lighthouse customers as test cases.
- What you can do after
- Apply controlled tests to evaluate go-to-market options and revise the business model.
AI-Powered Search
A/B testing is an experiment that compares two or more variants by randomly assigning users and measuring outcome differences.
- How it is used
- Validates deployed learning-to-rank models by comparing user outcomes across randomly assigned ranking variants.
- How deep it goes
- Defines random assignment, controls, outcomes, and simulation, then stops short of broader experimentation methodology.
- Examples
- Uses two ranking models, the query "transformers dvd," simulated sessions, and purchase counts as the comparison artifact.
- What you can do after
- Apply it via Python simulations to compare ranking models using tracked purchases and identify offline-to-production failures.
Effective Platform Engineering
Experimentation is the systematic evaluation of alternative designs or implementations under realistic conditions to inform decisions.
- How it is used
- Validates platform technology and feature decisions through realistic implementation, measured value, and early user feedback.
- How deep it goes
- Requires proof-of-concept implementation and alpha-user testing, emphasizing incremental evidence rather than abstract technology selection.
- Examples
- Uses actual-platform proofs of concept, Kubernetes service evaluations, and early tests of fundamental features with alpha users.
- What you can do after
- Evaluate platform technologies and feature designs before commitment, using implementation results and user feedback to guide the next decision.
Read how Effective Platform Engineering explains experimentation →
Architecture Modernization: Socio-technical alignment of software, strategy, and structure
A/B testing is an experiment that compares two variants of a product or experience with different users to evaluate their relative outcomes.
- How it is used
- Supports modernization and uncertain discovery by helping teams test technical and organizational approaches before committing.
- How deep it goes
- Explains trust, leadership support, throwaway code, reprioritization risks, and transitions, without detailing experimental measurement or implementation mechanics.
- Examples
- Uses CloudSuite’s workshops, hands-on coding, cross-team API modularization, Mob Programming, and big bet future cores as examples.
- What you can do after
- Apply an experimental mindset by creating space to try approaches, learn from results, and adjust direction as uncertainty decreases.
Think Like a Software Engineering Manager
Experimentation is the deliberate testing of ideas or changes to learn from observed results.
- How it is used
- Uses experimentation to compare market-specific changes and support iterative delivery through feedback and learning.
- How deep it goes
- Explains experimentation through workplace scenarios and iterative cycles, stopping short of implementation detail or statistical analysis.
- Examples
- Uses configurable return limits across marketplaces and partial product delivery followed by feedback.
- What you can do after
- Apply it via iterative delivery to test changes, recognize failures quickly, gather feedback, and learn from results.
Read how Think Like a Software Engineering Manager explains experimentation →
WordPress Power Toolkit: Harness AI to build next-level websites
A/B testing is an experimental technique that compares two or more variants to determine which performs better against a chosen outcome.
- How it is used
- Uses controlled page or campaign variations to improve marketing performance and understand visitor responses.
- How deep it goes
- Defines comparison and shows tool-assisted optimization, but stops short of explaining experimental design or statistical analysis.
- Examples
- Demonstrates testing pop-ups and forms, then pairs results with heatmaps, scroll maps, recordings, and engagement analysis.
- What you can do after
- Apply it via OptinMonster or Crazy Egg to compare variations and improve pages or campaigns.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains experimentation →
Delivering Machine Learning Projects: From design to deployment
An experiment is a structured investigation in which an uncertain proposition is tested through planned observations or actions.
- How it is used
- Guides machine-learning model development from exploratory investigation toward useful, reproducible results.
- How deep it goes
- Explains managed experimentation, upfront records, reproducibility, and data-leakage risks without detailing specific algorithms.
- Examples
- Uses lab-book records, documented model designs, validation-data optimization, and test-data protection as concrete practices.
- What you can do after
- Practice/perform structured model investigations that preserve evidence, reduce repeated work, and guard against misleading evaluation.
Read how Delivering Machine Learning Projects: From design to deployment explains experimentation →
Introduction to Generative AI, Second Edition
A/B testing is an experimental method that compares alternative versions by measuring their performance under controlled conditions.
- How it is used
- Measures how users respond to competing models or prompts in production.
- How deep it goes
- Explains the comparison setting and outcome metrics but stops short of experimental design or statistical procedures.
- Examples
- Compares two models or prompts using engagement, satisfaction, or other selected production metrics.
- What you can do after
- Evaluate user preferences empirically instead of relying only on a small sample of inspected responses.
Read how Introduction to Generative AI, Second Edition explains experimentation →
Causal AI
An A/B test is an experiment that compares outcomes under two assigned variants, commonly labeled A and B.
- How it is used
- Frames product or policy changes as interventions whose outcome differences should be estimated through random assignment.
- How deep it goes
- Explains assignment, intervention, group means, significance testing, costs, and causal complications, but defers detailed experimental diagnostics.
- Examples
- Uses an online game, side-quest engagement variants, in-game purchases, summary statistics, and a Python significance test.
- What you can do after
- Apply via Python by comparing randomized groups, summarizing outcomes, and testing their mean difference.
Related concepts
- control groupuses · A control group is a set of experimental units assigned to a baseline or comparison condition against which an intervention is evaluated.
- study designincludes · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons used to investigate a question and support valid conclusions.
- observational datacontrasts with · Observational data are measurements collected without the investigator assigning or controlling the conditions, treatments, or interventions being studied.
- simulationalternative to · A simulation is a modeled or controlled enactment of a system, process, or scenario used to examine or reproduce behavior under specified conditions.
- metricmeasures · A metric is a quantitative measure used to characterize or assess a property, condition, behavior, performance, or outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.