liveBook · Manning

Exposure logging

As explained in AI Model Evaluation

The recording of whether an assigned user or unit actually received or experienced the changed model output.

Exposure logging records whether an assigned user or unit actually received the changed model output. Assignment alone is insufficient: users may trigger a surface while the model fails, falls back, or serves cached results. Without exposure data, analysis can treat untreated users as treated and weaken the launch decision. section 6.3.5

Validate exposure before interpreting experiment metrics:

  • Record the assigned variant and whether the user actually saw the model output. section 6.3.5
  • Identify the served model version, feature set, ranking logic, prompt, or post-processing rule. section 6.3.5
  • Capture fallbacks to control models, cached results, default rules, or empty responses. section 6.3.5
  • Check overlapping experiments and consistent success and guardrail metric instrumentation. section 6.3.5

A lightweight exposure record

Extend a feature flag system to randomize model versions and log assignment, exposure, and results under a user or session ID. This creates a basic controlled experiment without requiring a full experimentation platform. Hosted services such as Optimizely, Eppo, Statsig, and Split.io can provide randomization, assignment, exposure logging, and metrics pipelines, although integration with model-serving and data systems still requires engineering work. section 6.7.2 section 6.3.5

How AI Model Evaluation covers it

How deep it goes
Explains logging requirements and serving failure modes, but stops short of specifying a schema or implementation code.
Examples
Uses user or session identifiers, feature-flag routing, model-version records, and hosted experimentation platforms as implementation examples.
What it assumes
Assumes identifiable users or sessions, traffic routing, metric collection, and awareness of model-serving fallbacks.
What you can do after
Apply via a feature flag system to route variants, log actual exposure, and interpret controlled experiment results more reliably.
How it connects
Connects exposure logging with A/B testing, randomization, feature flags, model serving, caching, fallbacks, and guardrail metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.