Production logging records enough system activity to show whether an ML system is behaving as expected and to trace what happened when it is not. It belongs alongside monitoring and alerting in the infrastructure that lets support groups accept and operate a release. section 9.5 section 9.11
Make logging an explicit production task:
- Record operational events, including database connections, user logins, data updates, and model decisions. section 9.5
- Capture functional as well as nonfunctional behavior. An ML system can stop making useful predictions when the world changes, even if its software still runs. section 9.5
- Set log density and retention for the application, accounting for the load of calculating metrics and the cost of storing logs. section 9.5
- Provide alerts and agree on support handover and maintenance documentation before release. section 9.5 section 9.11
Connecting project logs
A production setup can connect Splunk to logs from both the model server and the data subsystem while the team arranges for support groups to take over the system. section 9.10
For an inherited ML system, check first that activity logs exist, performance can be understood, and failures alert a support group. Missing logging and monitoring are technical debt: without logs, troubleshooting becomes harder, and failures may go unnoticed longer. section 10.2.2
