Chapter 14: Improving training with metrics and augmentation

Introduction

The chapter introduction explains why high overall accuracy did not mean the lung-nodule classifier was useful.

• Misleading accuracy with imbalanced data: Explain why the classifier appeared to perform well despite failing to detect nodules: the data was heavily skewed toward negative samples, so predicting non-nodule for everything produced a high percentage correct.
  Why it matters: It explains why the apparently good results from the previous chapter were not evidence that the classifier could find nodules.

Key terms: non-nodule, negative samples, training and validation sets

14.1 High-level plan for improvement

This section previews the chapter’s plan to improve the nodule classifier’s evaluation and training.

14.2 Good dogs vs. bad guys: False positives and false negatives

The section uses two guard dogs to explain classification outcomes and show why overall accuracy can hide serious mistakes in nodule detection.

• Positive and negative outcomes: Distinguish false positives, false negatives, true positives, and true negatives using the guard-dog metaphor and nodule-detection examples.
  Why it matters: These distinctions show whether a detector is raising unnecessary alarms or missing nodules.
• Limits of overall accuracy: Explain why a high percentage correct can be a poor assessment of the chapter 13 nodule detector, and relate its behavior to Dozer.
  Why it matters: A detector’s apparent success is misleading if it routinely misses potential cancers.

Key terms: false positive, true positive, false negative, true negative, percentage correct, false negatives, overall performance

14.3 Graphing the positives and negatives

This section uses a guard-dog diagram to show how ground-truth labels and a model’s thresholded predictions create four outcome groups whose counts can be used to evaluate performance.

• Labels, predictions, and quadrants: Explain how a human-assigned label and a thresholded prediction place each sample in one of four true/false positive/negative groups, and why the counts in those groups matter.
  Why it matters: These four groups are the starting point for describing a classifier’s errors more precisely than with a single overall score.
• Overlapping events and errors: Explain why a single bark-worthiness cutoff cannot perfectly separate positive and negative events when they overlap along the information available to the classifier.
  Why it matters: The overlap explains why choosing a cutoff involves errors rather than revealing a perfectly correct division.
• Model output and diagram axes: Describe how the diagram relates to the CT classifier: the model reduces high-dimensional sample information to one scalar output compared with a classification threshold, while the drawn Y-axis illustrates distinctions the model does not use.
  Why it matters: This keeps the visual analogy tied to what the CT classifier actually outputs and prevents treating the drawing’s second axis as another model output.

Key terms: human label, classification threshold, true/false positives/negatives, quadrants, bark-worthiness, X-axis, vertical bark-worthiness threshold, overlapped set of events, CT voxel values, nn.Linear layers

14.3.1 Recall is Chirpy’s strength

This section defines recall and uses Chirpy’s classification threshold to show how high recall can come with many false positives.

• Recall and missed positives: Explain that recall measures the share of actual positive events correctly identified: true positives divided by true positives plus false negatives. State that improving recall means missing fewer positives.
  Why it matters: Recall tells you whether the classifier is missing events you care about.
• Chirpy’s high-recall threshold: Describe how Chirpy moves her classification threshold far left to include nearly all positive events, yielding near-1.0 recall. Explain that this catches almost all robbers but also produces many false positives.
  Why it matters: Catching nearly everything of interest does not mean every alert is correct.

Key terms: recall, sensitivity, true positives, false negatives, classification threshold, high recall, false positives

14.3.2 Precision is Dozer’s forte

The section defines precision, shows how a high classification threshold favors it at the cost of missed events, and explains why precision and recall are both useful during training.

• Precision and positive predictions: Explain precision as the share of positive detections that are true positives, and relate higher precision to fewer false positives.
  Why it matters: Precision tells you how often the model is right when it flags something.
• Dozer’s high-threshold tradeoff: Describe how Dozer obtains high precision by moving his classification threshold to the right, and what he gives up as a result.
  Why it matters: Dozer illustrates why very trustworthy positive alerts can come with many missed events.
• Limits of individual metrics: State why precision and recall are useful during training without treating either as the sole measure of model performance.
  Why it matters: Tracking both avoids judging the model by just one kind of performance.

Key terms: precision, true positives, false positives, classification threshold, negative events, recall, training

14.3.3 Implementing precision and recall in logMetrics

The section updates logMetrics to count classification outcomes and report precision and recall during training.

• Classification outcome counts: Trace how logMetrics obtains the true-positive, true-negative, false-positive, and false-negative counts for nodule classification.
  Why it matters: These counts distinguish the kinds of mistakes hidden by an overall correctness figure.
• Precision and recall tracking: Explain why logMetrics adds precision and recall to its per-epoch output and what happens to the computed values.
  Why it matters: Tracking both metrics can reveal when training has gone wrong and help guide investigation.

Key terms: neg_correct, trueNeg_count, true positives, false positives, false negatives, precision, recall, logMetrics, metrics_dict

14.3.4 Our ultimate performance metric: The F1 score

This section explains why the F1 score is used to evaluate the classifier and how precision, recall, F1, and sample counts are added to training logs.

• Balanced precision and recall: Explain why F1 combines precision and recall more usefully than either metric alone or their arithmetic average. Describe how it favors improving the weaker rate rather than rewarding a highly skewed result.
  Why it matters: A score that rewards balance makes it harder to mistake a threshold-driven result for a useful classifier.
• Limits of simpler combinations: Distinguish the weaknesses of averaging, taking the minimum, and multiplying precision and recall as alternatives to F1.
  Why it matters: Comparing the alternatives shows what information a useful combined score must preserve.
• Expanded epoch metric logging: Describe which metrics and sample counts are added to the training and validation logging output.
  Why it matters: The expanded logs make the combined score and the underlying class results visible during training and validation.

Key terms: F1 score, precision, recall, harmonic mean, classification threshold, arithmetic mean, minimum, multiplication, geometric mean, logMetrics

14.3.5 How does our model perform with our new metrics?

The section interprets the model’s new metrics, explains its zero and undefined scores, and shows why they reveal a failure that earlier results obscured.

• Zero scores and undefined metrics: Explain why the model’s precision and recall are zero and why some F1 and precision calculations produce a division-by-zero warning.
  Why it matters: A zero or undefined score is an interpretable consequence of what the model predicted, not just a mysterious warning.
• Training and validation timing: Explain why a few negative training samples were flagged as positive even though the validation results flagged nothing as positive.
  Why it matters: The training and validation counts reflect predictions made at different points in training.
• More revealing performance metrics: Explain why the poor new scores are useful despite looking much worse than the earlier result.
  Why it matters: An honest measure of poor performance is more useful than a reassuring number that hides it.

Key terms: positive samples, precision, recall, F1 score, truePos_count, falsePos_count, training results, validation results, epoch, first batch

14.4 What does an ideal dataset look like?

The section explains why the current classifier needs more separable predictions and why balancing its training data is a practical step toward that goal.

• Separable Classes and Thresholds: Explain why changing the classification threshold alone cannot fix the current results, and describe the separation the authors want instead.
  Why it matters: It explains why adjusting the cutoff is not enough and why the authors turn to the data.
• Training Class Balance: Explain why the authors propose changing the training class balance despite wanting the finished model to handle imbalanced data.
  Why it matters: It distinguishes a practical choice about training data from the imbalance the finished classifier must handle.

Key terms: classification threshold, label threshold, positive and negative classes, separable, class balance, negative samples, positive samples, training data, data imbalance

14.4.1 Making the data look less like the actual and more like the “ideal”

Explains why rare positive nodule samples cause training to favor negative predictions, then shows how to balance and intermix training samples in LunaDataset while leaving validation representative of real data.

• Imbalance and negative-only predictions: Explain why the original LUNA training mix can push an initially random model toward predicting negative for nearly every candidate.
  Why it matters: High apparent accuracy can hide a model that has learned to answer negative rather than distinguish nodules.
• Balanced training and realistic validation: Justify intermixing roughly equal numbers of positive and negative training samples while keeping validation imbalanced.
  Why it matters: Training can emphasize rare nodules without making validation results describe an artificially balanced world.
• Ratio-based LunaDataset balancing: Describe how LunaDataset presents a chosen ratio of negative to positive training samples despite having few distinct positives, and why its epoch length can be set independently of the raw sample count.
  Why it matters: This makes the desired training mix possible without collecting more positive nodules.
• Dataset shaping versus samplers: Distinguish using a DataLoader sampler to reshape sampling from implementing the balancing inside a dataset the project controls.
  Why it matters: The choice avoids depending on hidden dataset internals when the project can modify its own dataset.

Key terms: positive samples, negative samples, degenerate behavior, batch size, balanced, training, validation, discrimination, LunaDataset, ratio_int

14.4.2 Contrasting training with a balanced LunaDataset to previous runs

The section compares balanced and unbalanced training, shows why class imbalance still limits practical performance, and contrasts later validation results with training results.

• Balanced Training Tradeoff: Describe how the balanced run improved positive-sample classification while slightly reducing negative-sample classification, and why that made the model more useful for screening.
  Why it matters: The balanced run is an improvement, but its value depends on which mistakes it now makes.
• Class Imbalance and False Positives: Explain why high negative-sample accuracy can still produce too many false positives when negative samples vastly outnumber positives.
  Why it matters: A percentage that looks small can represent many errors when it applies to the much larger class.
• Diverging Training and Validation Results: Describe what happened to positive validation results over more epochs and contrast them with the training results, without assuming that more training fixed the missed positives.
  Why it matters: Strong training numbers alone do not show that additional training improved validation performance.

Key terms: --balanced, val_neg, val_pos, class imbalance, false positives, negative validation samples, positive validation samples, epochs, validation, training set

14.4.3 Recognizing the symptoms of overfitting

This section diagnoses overfitting in the nodule classifier by comparing training and validation loss for positive samples, then explains why overall loss hides the problem.

• Diverging positive sample losses: Explain why nearly zero training loss alongside increasing validation loss for positive samples signals overfitting, and why continuing training is unlikely to help.
  Why it matters: The two loss curves show when better training predictions no longer mean better performance on unseen data.
• Misleading overall loss: Explain why overall loss appears healthy even though positive validation loss reveals overfitting, and contrast the negative-sample loss.
  Why it matters: Checking loss by class exposes a failure that the aggregate metric conceals.
• Memorization of scarce positives: Describe why repeating the small set of positive training samples does not prevent memorization, and how that affects validation predictions.
  Why it matters: Repeated exposure to the same positives can improve training predictions without supplying new examples to generalize from.

Key terms: overfitting, training loss, validation loss, positive samples, overall loss, positive loss, negative loss, validation set, not balanced, positive training set

14.5 Revisiting the problem of overfitting

The section explains overfitting as learning training-set specifics instead of properties that generalize to unseen samples.

• Generalization versus overfitting: Explain that training aims to capture general properties of a class that help predict unseen samples, whereas overfitting occurs when a model learns properties specific to the training set and loses that ability.
  Why it matters: This distinction explains why learning the training data is not, by itself, the goal.

Key terms: general properties, training set, overfitting, generalize

14.5.1 An overfit face-to-age prediction model

A face-to-age example shows how memorizing training images differs from learning age cues, and why too little data relative to model capacity encourages overfitting.

• Age cues versus memorization: Distinguish a face-to-age model that generalizes from one that overfits by remembering particular people.
  Why it matters: This contrast makes clear why remembering training examples is not the same as making useful predictions about new people.
• Training data and model capacity: Explain why a model is likely to overfit when its capacity to memorize hard training examples is high relative to the available training data.
  Why it matters: The balance between data and memorization capacity helps explain why a model can fit training examples without learning a general rule.

Key terms: age signifiers, overfit model, identifying details, overfitting, training samples, model capacity, parameters

14.6 Preventing overfitting with data augmentation

The section explains how data augmentation can reduce overfitting and why useful alterations matter more than a large number of trivial variations.

• Augmentation and generalization: Explain how synthetic alterations can expand the effective training set while keeping samples representative of their original class, making memorization harder and encouraging generalization, especially when data is limited.
  Why it matters: Augmentation is useful because more effective training examples can push a model to learn general patterns rather than memorize samples.
• Useful versus trivial alterations: Distinguish the face-image example's random corner-pixel changes from left-to-right flips, explaining why the smaller set of mirrored images is more useful for training.
  Why it matters: A vast collection of negligible changes may help less than a modest number of meaningful variations.

Key terms: data augmentation, synthetic alterations, generalization, training set, face-to-age prediction model, red channel, flipping the image left to right, mirrored image

14.6.1 Specific data augmentation techniques

This section implements five CT-candidate augmentations and explains how to keep transformed samples representative while producing varied training examples.

• Representative CT transformations: Justify choosing augmentations that change a training sample without making it unrepresentative. Explain why left-right and front-back flips are considered reasonable for these CT candidates, while an up-down flip rests on an assumption that would need expert confirmation for clinical use.
  Why it matters: A transformation that creates unrealistic examples can undermine training rather than improve it.
• Affine spatial resampling: Describe how the CT chunk is converted to a tensor and resampled with an affine transformation matrix, affine_grid, and grid_sample. Explain how mirroring, random offsets, and slight scaling alter the sample, including the possible interpolation blur and repeated edge voxels.
  Why it matters: The shared resampling approach explains both how several augmentations are produced and why their images may show artifacts.
• CT rotation axis constraint: Explain why the section confines CT-candidate rotations to the X-Y plane instead of treating all three axes alike.
  Why it matters: Respecting the scan’s unequal axis resolution avoids creating misleading rotated examples.
• Noise augmentation tradeoff: Distinguish adding Gaussian noise from the spatial augmentations and explain why its amount must be limited.
  Why it matters: More variation is not automatically helpful if it obscures what the model needs to classify.
• Fresh random augmentation: Explain why caching must precede augmentation and why each dataset access applies the chosen augmentations randomly, including the possibility of no flip.
  Why it matters: Reapplying random changes lets the same cached candidate supply different training examples over time.

Key terms: representative nature, mirroring, left-right, front-back, index-axis, getCtAugmentedCandidate, affine transformation matrix, affine_grid, grid_sample, trilinear interpolation

14.6.2 Seeing the improvement from data augmentation

This subsection compares validation results from different augmentation schemes and explains why the project continues with full augmentation despite its trade-offs.

• Augmentation comparison setup: Describe how the training runs compare individual augmentation types with all types combined, and how their validation results are viewed in TensorBoard.
  Why it matters: Comparing validation runs side by side makes the effects of augmentation easier to judge.
• Combined augmentation results: Explain why the fully augmented model is judged favorably even though it gets more answers wrong overall.
  Why it matters: Overall correctness can hide an improvement in finding the nodules that matter most.
• Recall driven model choice: Justify continuing with full augmentation rather than the rotation-only model, and state how F1 is still used.
  Why it matters: The best model choice depends on the project's priority, not just which run has the highest F1 score.

Key terms: augmentation_dict, TensorBoard, validation data, fully augmented, positive candidate samples, recall, overfitting, rotation-augmented, precision, F1 score

14.7 Conclusion

The conclusion emphasizes trustworthy model evaluation and representative synthetic training samples as two ways to improve a classifier.

• Reliable Model Performance Evaluation: Explain why evaluating a model requires more than a potentially misleading performance measure and why an intuitive grasp of the factors behind evaluation helps reveal mistakes.
  Why it matters: A performance result is useful only if the way you evaluate it supports the conclusion you draw.
• Representative Samples from Scarce Data: Explain why synthesizing representative training samples is useful when a data source is insufficiently populated.
  Why it matters: Not having enough training examples is a common problem, so creating representative ones can be valuable.

Key terms: model’s performance, evaluation, data sources, representative training samples

14.8 Exercises

14.9 Summary