liveBook · Manning

Randomization

As explained in AI Model Evaluation

The assignment of experiment units to variants by chance according to a specified allocation procedure.

Randomization assigns experiment units to control or treatment variants by chance, creating a controlled comparison for measuring a model’s product impact. Choose the unit deliberately—user, account, session, or device—and verify that observed traffic matches the planned allocation. section 6.3.5

Assignment is not exposure. A treatment-assigned user may never reach the model surface, may receive a fallback, or may see cached output. Log both assignment and actual exposure before interpreting outcome metrics. section 6.3.5

Validate a randomized experiment before trusting its results:

  • Confirm that users were assigned randomly using the intended unit and that the traffic split matches the planned allocation. section 6.3.5
  • Record variant assignment, model version, feature set, ranking logic, post-processing, exposure, fallbacks, and cached responses. section 6.3.5
  • Check overlapping experiments, metric consistency, pre-planned segments, and decision rules before examining results. section 6.3.5

A lightweight model experiment

Extend an existing feature-flag system to randomize exposure to model versions, associate assignments and results with a user or session ID, and measure key metrics over time. Hosted services can provide randomization, assignment, exposure logging, and metrics pipelines when building infrastructure in-house is impractical. section 6.7.2

How AI Model Evaluation covers it

How deep it goes
Explains assignment units, allocation checks, exposure logging, and implementation failure modes, but stops short of statistical test formulas.
Examples
Uses user or session assignment, 50/50 traffic allocation, model-version exposure, cached responses, and feature-flag-based experiments.
What it assumes
Assumes identifiable users or sessions, traffic routing, metric collection, and a feature-flag system, while treating statistical complexity as nonessential initially.
What you can do after
Operate/configure controlled experiments by routing traffic, logging assignment and exposure, and validating instrumentation before reading metrics.
How it connects
Connects randomization to A/B testing, feature flags, exposure logging, model serving, fallbacks, metrics, and overlapping experiments.

Also explained in

Compare how 7 books explain randomization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.