liveBook · Manning

Unit of randomization

As explained in AI Model Evaluation

A unit of randomization is the entity, such as a user, session, or account, that is independently assigned to an experiment variant.

A unit of randomization is the entity independently assigned to an experiment variant, such as a user, session, or account. Choosing it determines how consistently people experience the treatment and how interpretable the measured effect will be. section 6.3.2

Define the unit alongside the other mechanics that connect a hypothesis to rollout:

  • Specify which entity receives assignment: user, session, account, or another operational unit. section 6.3.2
  • Keep the experience consistent when consistency is needed for interpretation; user-level assignment gives one user the same recommendation experience during the test. section 6.3.6
  • Interpret assignment with exposure rules, exclusion criteria, success metrics, and guardrail metrics rather than treating the unit choice as sufficient evidence of impact. section 6.3.2 section 6.3.6

Movie recommendation test

A recommendation experiment uses user-level randomization. Active users who watched at least one title in the past 14 days enter the test, while users already enrolled in another recommendation experiment on the same surface are excluded. Each user consistently receives the control model, Treatment A with conservatively weighted contextual features, or Treatment B with stronger weighting on recent searches and session context. section 6.3.6

How AI Model Evaluation covers it

How deep it goes
Explains assignment choices and user-level consistency, extending into exposure rules but not statistical derivation.
Examples
Uses user, session, and account choices, then applies user-level assignment to a movie recommendation test.
What you can do after
Apply it by selecting an assignment entity and checking consistent exposure against success and guardrail metrics.
How it connects
Connects experimental design to variants, metrics, segmentation, guardrail metrics, and operational rollout.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.