How deeply does Machine Learning System Design cover model opacity?

Defines opacity and connects it to risk modeling, stakeholder alignment, and requirements, without explaining interpretability techniques.

Model opacity

As explained in Machine Learning System Design

The property of a machine-learning model whose internal reasoning or decision process is difficult for observers to understand.

Model opacity is the difficulty observers face when trying to understand a machine-learning model’s internal reasoning or decisions. It makes system design a responsibility problem, not only a prediction problem: teams must account for limitations, consequences, and risks that may spread across stakeholders. section 2.3

Treat opacity as part of the system’s nonfunctional requirements:

Define a risk model that answers what the system must protect against; “incorrect prediction” is too vague without detailed worst-case scenarios. section 2.3
Discuss risks across teams and departments rather than assuming professional organizations are immune to opaque-system failures. section 2.3
Separate functional requirements, which describe value and user flow, from nonfunctional requirements, which shape how the system should work. section 2.3

Dynamic pricing

A dynamic pricing algorithm optimized revenue, margin, or traffic under constraints and could operate with user-level atomicity. That flexibility increased the need to reason explicitly about risks and system design. section 2.3

How Machine Learning System Design covers it

liveBook manning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

training data transparencycontrasts with · Training data transparency is the disclosure of a model’s training dataset contents, provenance, composition, limitations, and intended uses.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.