liveBook · Manning

Context

As explained in AI Model Evaluation

A contextual feature is an input variable describing the circumstances surrounding a user's current interaction or request.

Context describes the circumstances surrounding a user’s interaction or request. Contextual features can include device type, time of day, location, recent activity, and query intent. They help distinguish situations in which the same recommendation or ranking may produce different outcomes. section 5.2.2 section 5.7.4

Context can be both a model input and an evaluation input. A recommender might use device type, time of day, and recent searches to improve ranking; evaluators can also use logged context to compare outcomes under different conditions. section 6.3.6 section 5.5.3

For counterfactual evaluation, log enough information to reconstruct each decision:

  • Capture the context available at decision time, including user, environmental, and session details that may affect behavior. section 5.2.1 section 5.5.3
  • Record the action taken, its probability under the logging policy, and the observed outcome alongside the context. section 5.2.1
  • Log consistently across actions and user groups. Incomplete or uneven context can skew comparisons. section 5.5.4 section 5.2.2
  • Protect sensitive data: consider regulatory requirements and anonymize where possible. section 5.7.4

Missing context can make a policy appear effective because of when or where it was used, or conceal poor performance for particular query types. Accurate logs, sufficient overlap between policies, and the relevant decision-time context are all needed for credible counterfactual estimates. Treat those estimates as evidence for deciding what to test, not as a substitute for an A/B test. section 5.7.4 section 5.2.1 section 5.2.2

Testing a contextual movie recommender

A streaming platform adds device type, time of day, and recent searches to an existing model. An A/B test checks whether click-through rate and watch time improve without increasing latency or reducing satisfaction. It also checks for lower-quality recommendations among important user groups. Separately, an off-policy analysis may reveal that a viewing-time gain is stronger on weekends than at other times. section 6.3.6 section 5.8.3

Context can also mean the evidence supplied to an LLM judge, rather than a user-interaction feature. For example, verifying a recommendation explanation requires the relevant user history, recommendation list, and metadata; without that evidence, narrow the judgment to what can be assessed. section 9.7.4

How AI Model Evaluation covers it

How deep it goes
Explains what to log, why missing context distorts comparisons, and the assumptions needed for counterfactual estimates.
Examples
Illustrates device and time features in movie recommendations, weekend viewing differences, and informational versus transactional search queries.
What you can do after
Evaluate and understand whether logged context supports meaningful policy comparisons and whether contextual features improve recommendations without harming guardrails.
How it connects
Connects context to recommendation systems, counterfactual evaluation, propensity scores, policy value, and A/B testing.

Also explained in

Compare how 30 books explain context →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.