liveBook · Manning

Multi-criteria decision-making

As explained in Delivering Machine Learning Projects: From design to deployment

Multi-criteria decision making is a decision-analysis approach for evaluating alternatives against multiple criteria, often with explicit weights or tradeoffs.

Multi-criteria decision making (MCDM) selects among models by combining results from tests that measure different, sometimes incomparable, aspects of performance. The aggregation method can change the selection, so its rationale should be explicit and documented. section 8.3.3 section 8.6

Use MCDM as an explicit selection artifact:

  • List the tests that matter, including separate sets for hard-but-important cases and run-of-the-mill data. section 8.3.3
  • Choose an aggregation method, such as weighted scores, rankings, or a Pareto front, based on the project’s decision needs. section 8.3.3 section 8.6
  • Record the weights, trade-offs, assumptions, and unresolved judgment behind the final choice. section 8.3.3 section 8.6

Hard-but-important versus routine data

Suppose a test set contains 200 hard-but-important examples and 200,000 routine examples. A 50/50 aggregate can make each hard-but-important result worth 1,000 routine results, ensuring routine performance does not erase the targeted test’s purpose. The weighting remains a judgment: a small change in hard-case performance may outweigh a larger routine-data improvement. section 8.3.3

Table 8.2 Comparing five models with aggregate rankings (view table figure)

Model

hbi (raw)

hbi (rank)

rotm (raw)

rotm (rank)

Aggregate

Rank

A

171/200

3

175342/200000

2

5

3

B

167/200

4

172811/200000

4

8

4

C

132/200

5

135241/200000

5

10

5

D

173/200

2

181122/200000

1

3

1

E

190/200

1

175301/200000

3

4

2

Compare five models using raw test results, per-test ranks, aggregate rankings, and final selection order. section 8.3.3

How Delivering Machine Learning Projects: From design to deployment covers it

How deep it goes
Explains weighted aggregation, rankings, and Pareto fronts, while stopping short of prescribing one universally correct method.
Examples
Uses hard-but-important and run-of-the-mill test sets, weighted scores, and five-model aggregate rankings.
What it assumes
Assumes multiple model tests and performance results, while treating aggregation and selection judgment as the central problem.
What you can do after
Evaluate competing models with explicit aggregation choices and document the reasoning behind the final selection.
How it connects
Connects model selection with rankings, Pareto fronts, cross-validation, A/B testing, multi-armed bandits, and qualitative judgment.
Cover of Delivering Machine Learning Projects: From design to deployment
liveBookmanning.com
Read Delivering Machine Learning Projects: From design to deployment on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.