Offline evaluation in Delivering Machine Learning Projects: From design to deployment

Q: How deeply does Delivering Machine Learning Projects: From design to deployment cover offline evaluation?

Defines data splits, unseen testing, leakage avoidance, and cross-validation, but does not detail online testing.

Offline testing evaluates a model using previously collected or static data in a non-live environment before deployment.

Offline evaluation tests a model against data collected and reserved for evaluation rather than live traffic. It supports model testing and selection before production, while requiring documented, reproducible processes and a testing scope suited to the project. section 8.1 section 8.2.1

Implement offline evaluation as a controlled data and modeling workflow:

Partition available data into training and test data; a simple example uses 70% for training and 30% for testing. section 8.2.1
Divide the training data into data shown to the algorithm and a validation or hold-out set used during development. section 8.2.1
Reserve test data that was unavailable during modeling so iterative decisions cannot unconsciously optimize against it. section 8.2.1
Use cross-validation when appropriate by rotating disjoint partitions through training and testing, then compare the resulting evaluations. section 8.2.1
Record the selected process and its results so the evaluation can be reproduced and its effort justified. section 8.1

Ten-fold cross-validation

Divide the dataset into ten disjoint partitions. Train on nine partitions and test on the remaining partition, repeating the process until each partition has served as the test set. section 8.2.1

Table 8.1 Generating a cross-validation set (view table figure)

	P1	P2	P3	P4	P5	P6	P7	P8
Set 1	Test	Train	Train	Train	Train	Train	Train	Train
Set 2	Train	Test	Train	Train	Train	Train	Train	Train
Set 3	Train	Train	Test	Train	Train	Train	Train	Train
Set 4	Train	Train	Train	Test	Train	Train	Train	Train
Set 5	Train	Train	Train	Train	Test	Train	Train	Train
Set 6	Train	Train	Train	Train	Train	Test	Train	Train
Set 7	Train	Train	Train	Train	Train	Train	Test	Train
Set 8	Train	Train	Train	Train	Train	Train	Train	Test

Use the table to inspect how each partition becomes the test set while all other partitions remain training data. section 8.2.1

How Delivering Machine Learning Projects: From design to deployment covers it

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Delivering Machine Learning Projects: From design to deployment. The section links point to the parts of the book it draws on.