liveBook · Manning

Log

As explained in AI Model Evaluation

Logging is the practice of recording detailed information about system behavior, user interactions, and outcomes.

A log records system decisions, user interactions, and outcomes. Structured logs can become inputs to offline evaluations, help diagnose model behavior, and make online results interpretable. A click is useful evidence, for example, but it is a proxy for preference: a user may click a movie because it was prominently placed rather than because it was the best recommendation. section 2.2.1 section 6.4.3

Choose what to log for the evaluation question. Diagnostic evaluations need more detail than aggregate accuracy or precision: user and session context, recommendation attributes, and outcomes can reveal whether a system underserves particular groups. section 3.7.1 section 5.5

Listing 3.5 detailed logging example
{
  "user_id": "abc123",
  "age_group": "35-44",
  "session_id": "sess-789",
  "timestamp": "2025-07-12T14:03:21Z",
  "recommendations": [
    {
      "movie_id": "m001",
      "genre": "Action",
      "popularity_score": 0.92,
      "novelty_score": 0.15,
      "position": 1
    },
    {
      "movie_id": "m002",
      "genre": "Documentary",
      "popularity_score": 0.32,
      "novelty_score": 0.67,
      "position": 2
    }
  ],
  "user_clicked": "m002"
}

This diagnostic event connects a movie recommendation list and its positions and attributes to a user's click and session context. section 3.7.1

For counterfactual evaluation, preserve the decision as it appeared at the time:

  • Record the context available to the system, the action selected, and the observed outcome. Capture the candidate set considered; in large action spaces, logging every conceivable action may be impractical. section 5.2.1 section 5.5.1
  • Record the probabilities or scores needed by the planned estimator, along with the policy version. Probabilities enable methods such as inverse propensity weighting. section 5.5.2 section 5.5.5
  • Join downstream outcomes back to the decision and monitor for missing fields, schema changes, or delayed joins. Log consistently across actions and contexts to reduce skew. section 5.5.5 section 5.5.4

A search-ranking decision

For a wireless-earbuds query, a search log can capture the query and device, the products considered for the first position, their probabilities, the selected ranking, and subsequent clicks. These fields support questions about an alternative ranking. They do not guarantee a reliable answer: the logs must be accurate, capture important context, and provide sufficient overlap between the logged and candidate policies. section 5.5.5 section 5.2.1

In an A/B test, distinguish assignment from actual exposure. Record the model version, output, fallback path, latency, errors, and user outcomes: an assigned user might receive a cached or fallback ranking rather than the intended model output. For blended systems, component scores, active variants, and a decision trace help attribute changes to the right part of the pipeline. section 6.4.3 section 8.7.1

Log intentionally rather than indiscriminately. Rich context supports evaluation, but storage, privacy, and operational costs limit what is wise to collect. Anonymize sensitive data where possible and address regulatory requirements. To protect responsiveness, consider asynchronous logging or a dedicated logging service instead of writing everything in the request path. section 5.5 section 5.7.4 section 5.10.1

How AI Model Evaluation covers it

How deep it goes
Provides field-level logging checklists, quality checks, and latency trade-offs for evaluation workflows.
Examples
Uses a movie recommendation diagnostic schema and a search-ranking event with candidate products and action probabilities.
What you can do after
Operate/configure logging to capture evaluation-relevant decisions, outcomes, and serving paths while monitoring data quality.
How it connects
Connects logging to counterfactual evaluation, inverse propensity weighting, A/B testing, and attribution.

Also explained in

Compare how 21 books explain log →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.