liveBook · Manning

Monitoring

As explained in Machines that Think

Monitoring is the ongoing observation and tracking of system behavior after deployment to detect changes or emerging problems.

Monitoring is the ongoing observation of deployed system behavior. It helps detect emerging failures and track whether unequal patterns appear, disappear, or worsen as systems are used. section 6.3.5 section 7.4.3

Behavior may change when a model is connected to retrieval, tools, user feedback, or new workflows, or when the model, data, or application is updated. Monitoring supports recurring review and adjustment, but cannot eliminate variability; reliability also depends on the surrounding system. section 6.3.5 section 7.4.3

Figure 7.7 Techniques for exposing bias. Bias testing compares original and modified inputs to check whether small changes produce unjustified differences in output. Subgroup evaluation compares outcomes across relevant groups or contexts to detect disparities hidden by average performance. Benchmark evaluation uses standardized test sets to measure known failure patterns, while still requiring caution because benchmarks only reveal the behaviors they are designed to test.

Bias testing, subgroup evaluation, and benchmarks offer ways to examine behavior, though each reveals only the patterns it is designed to test. section 7.4.3

How Machines that Think covers it

How deep it goes
Describes systematic output review, stress-testing, and recurring safeguard adjustments, but does not specify monitoring metrics or implementation procedures.
Examples
Uses output review and stress-testing, and tracks disparities after systems connect to retrieval, tools, feedback, or new workflows.
What you can do after
Diagnose emerging failures and track whether unequal behavior appears, disappears, or worsens after deployment.
How it connects
Links monitoring to red teaming, auditing, system safeguards, and reliability.

Also explained in

Compare how 51 books explain monitoring →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.