Chapter 18: Best practices for the real world

Introduction

This introduction explains the gap between the book’s small-scale examples and the demands of real-world, state-of-the-art modeling.

• Small-scale to real-world gap: Explain why being able to train the kinds of models shown in the book does not, by itself, mean those models are ready to achieve state-of-the-art results on new problems.
  Why it matters: Training a working model is not the same as getting the strongest possible result on a new problem.

Key terms: small scale, best achievable performance, state-of-the-art results

18.1 Getting the most out of your models

The section contrasts trying architecture configurations blindly for an adequate model with the aim of building a stronger one.

• Blind Architecture Trial Results: Explain when blindly trying different architecture configurations may be sufficient.
  Why it matters: The result you need determines whether an adequate model is enough.

Key terms: architecture configurations

18.1.1 Hyperparameter optimization

This section explains how to search for model hyperparameters with KerasTuner, evaluate configurations responsibly, and turn search results into a final model.

• Hyperparameters versus trained parameters: Distinguish architecture-level hyperparameters from model parameters learned by backpropagation, and explain why the section favors systematic search over relying only on intuition and manual tweaking.
  Why it matters: It clarifies which model decisions training learns and which decisions a search must evaluate.
• Search loop and constraints: Trace the automatic hyperparameter search loop and explain why it cannot generally be optimized like model weights.
  Why it matters: It explains both what a tuner does and why each experiment can be expensive.
• KerasTuner search configuration: Describe how KerasTuner defines a search space and how a tuner runs trials, including the model-building function or HyperModel alternative.
  Why it matters: It connects the search idea to the model definition and trials that KerasTuner actually runs.
• Validation metrics and overfitting: Explain why tuning uses validation data rather than test data, how metric direction is handled, and why extensive searching can overfit the validation set.
  Why it matters: A promising search result is only useful if its performance estimate remains trustworthy.
• Final model after search: Describe how the best configurations can be used for final training, including choosing an epoch count before incorporating validation data.
  Why it matters: It turns a selected configuration into a model trained on the available development data without skipping the epoch decision.
• Deliberate search space design: Justify why a useful search space still requires architecture judgment rather than making every possible design choice tunable.
  Why it matters: A well-designed search explores plausible models within a practical training budget.

Key terms: hyperparameters, parameters, backpropagation, automatic hyperparameter optimization, hyperparameter space, validation performance, gradient-free optimization, random search, Bayesian optimization, KerasTuner

18.1.2 Model ensembling

This section explains how combining classifier predictions can improve results, especially when the models make different kinds of errors.

• Diversity in model ensembles: Explain why an ensemble benefits from models that are both strong and different, and why even a weaker model can help if it contributes a distinct perspective.
  Why it matters: Choosing complementary models can matter more than adding another copy of your strongest model.
• Averaging classifier predictions: Describe simple and weighted averaging of classifier predictions at inference time, including when equal averaging can fail and how weights are chosen.
  Why it matters: Weighting prevents a substantially worse classifier from having the same influence as a better one.

Key terms: model ensembling, diversity, bias, architectures, average, weighted average, validation data, inference time

18.2 Scaling up model training with multiple devices

The section explains why faster model training improves deep learning work by allowing more cycles of experimentation and refinement.

• Training speed and progress: Explain how training speed affects the loop of progress: once coding experiments is no longer the bottleneck, faster training lets you run and analyze more experiments, refining ideas more often.
  Why it matters: Faster training gives you more chances to test results and improve an idea.

Key terms: loop of progress, refinement cycles, training infrastructure, bottleneck

18.2.1 Multi-GPU training

This section explains how multi-GPU training distributes work by replicating a model, splitting a model, or combining both approaches.

• Data parallel training: Trace how data parallelism splits a batch among copies of the full model and combines their results for inference or training.
  Why it matters: It explains how more GPUs can process more examples while keeping a single, consistent model.
• Model parallelism purpose: Distinguish model parallelism from data parallelism and explain when the section says model parallelism is needed.
  Why it matters: It clarifies which form of parallelism addresses a model that cannot fit on one GPU.
• Model partitioning strategies: Describe horizontal and vertical partitioning and the trade-off the section identifies between them.
  Why it matters: The two ways of splitting a model have different implications for communication and device use.
• Combined parallelism layout: Explain how model and data parallelism can be used together in the section’s eight-device example.
  Why it matters: Combining the approaches lets device groups hold a large model while different groups work on different data.

Key terms: data parallelism, replica, sub-batches, synchronous training, model parallelism, sharding, partitioning, horizontal partitioning, vertical partitioning, communication overhead

18.2.2 Distributed training in practice

Shows how to configure data and model parallelism with Keras on JAX, and how to keep the input pipeline efficient during distributed training.

• JAX data parallel setup: Describe how to enable data parallelism before building a Keras model, optionally selecting specific devices, and explain why speedup depends on batch size and distribution overhead.
  Why it matters: The configuration is simple, but its benefit depends on giving each GPU enough work.
• Device mesh axes: Explain how a device mesh organizes GPUs for a mixture of data and model parallelism, using the section’s 2 × 4 example.
  Why it matters: The mesh makes it possible to describe how devices share both data and model computation.
• Variable layout decisions: Distinguish replication from sharding and describe how a LayoutMap specifies those choices for model variables.
  Why it matters: Variable layouts tell Keras which values devices share and which pieces they hold separately.
• Model sharding workflow: Describe the section’s rule of thumb for laying out a simple model’s variables, when to set the distribution configuration, and how training proceeds afterward.
  Why it matters: Setting the layout before variables are created lets the existing model and training code run with the intended distribution.
• Distributed input pipeline: State the recommended data format and prefetching practice for distributed training.
  Why it matters: An efficient input pipeline helps prevent data delivery from slowing distributed training.

Key terms: data parallelism, DataParallel, global batch size, local batch size, device mesh, model parallelism, axis_names, LayoutMap, variable path, replicate

18.2.3 TPU training

This section explains why TPU training is useful and how to configure Keras training, avoid data-loading bottlenecks, and keep TPU cores busy.

• TPU benefits and access: Explain that TPUs are specialized chips for deep learning, available through Google Colab and Google Cloud, and why their speed can justify the extra setup.
  Why it matters: It explains when the extra effort of using a TPU may be worthwhile.
• Keras TPU distribution setup: Describe how to start using TPU cores in a TPU-enabled notebook when running Keras with the JAX backend, including when to set the distribution.
  Why it matters: The distribution setting is the section’s concrete step for putting Keras training on TPU cores.
• TPU data loading bottleneck: Explain why reading data from Google Cloud Storage can limit TPU training and how caching a small dataset addresses it.
  Why it matters: Faster computation will not help fully if the TPU must wait for data.
• Large batches and learning rate: Explain why keeping TPU cores busy may require very large batches and why the section recommends increasing the learning rate when using them.
  Why it matters: Batch size affects both TPU utilization and how the optimizer should update weights.
• Step fusing for utilization: Explain how step fusing can keep reasonably sized batches while improving TPU utilization, and state how the section enables it in Keras.
  Why it matters: It offers a way to use an underutilized TPU effectively without resorting to enormous batches.

Key terms: ASIC, Tensor Processing Unit (TPU), Google Colab, Google Cloud, JAX backend, keras.distribution.set_distribution, DataParallel, ModelParallel, Google Cloud Storage (GCS), dataset.cache()

18.3 Speeding up training and inference with lower-precision computation

Introduces a potential way to speed up model training and inference, then points ahead to an explanation of precision.

18.3.1 Understanding floating-point precision

This section explains how bit count limits floating-point resolution, why representable values are unevenly spaced, and what practical resolution different precisions provide.

• Finite floating-point resolution: Explain why a computer cannot represent the continuous real-number line exactly and how the number of bits used to store a floating-point number determines its precision.
  Why it matters: It explains why changing a model’s numerical precision changes the values it can represent.
• Uneven floating-point spacing: Explain why representable floating-point numbers are not evenly spaced and how the resulting representation error changes with a number’s magnitude.
  Why it matters: Numerical error depends not just on the chosen format but also on the magnitude of the value being represented.
• Practical precision thresholds: State the approximate smallest distance between arbitrary numbers that the section says can be safely processed in half, single, and double precision.
  Why it matters: These approximate distances make the trade-off between floating-point formats concrete.

Key terms: precision, float16, float32, float64, half precision, single precision, sign, mantissa, exponent, representable values

18.3.2 Float16 inference

This section explains how 16-bit computation can speed up model inference and compares float16 with bfloat16.

• Lower precision inference speedups: Explain why 16-bit operations can speed up inference on GPUs and TPUs, and describe when to set Keras’s default precision to float16. Contrast this with the float32 precision used by earlier models, including for small training updates.
  Why it matters: It connects a simple inference setting to the hardware advantage that makes predictions faster.
• Bfloat16 range resolution tradeoff: Distinguish bfloat16 from float16 in numerical range and resolution, and explain why it is worth testing both on the intended device.
  Why it matters: The fastest 16-bit format can depend on the hardware, and the two formats have different numerical tradeoffs.

Key terms: float32, float16, model inference, model.predict(), bfloat16, exponent bits, mantissa bits

18.3.3 Mixed-precision training

Explains why training needs mixed precision and how Keras combines 16-bit computation with 32-bit weight updates.

• Mixed Precision Tradeoff: Explain why training entirely in 16-bit precision can be unstable and how mixed-precision training retains speed without meaningfully affecting model quality.
  Why it matters: The speed benefit is useful only if training remains numerically stable.
• Keras Layer Precision Roles: Describe how Keras layers use compute_dtype and variable_dtype during mixed-precision training, including how gradients reach the stored weights.
  Why it matters: Separating computation precision from stored-weight precision explains how Keras makes mixed-precision training practical.
• Float32 Layer Exceptions: Describe when and how to keep a particular layer in float32 despite mixed precision.
  Why it matters: Mixed precision need not force numerically sensitive layers to use float16.

Key terms: mixed-precision training, float16, gradient updates, numerical stability, compute_dtype, variable_dtype, float16 gradients, float32 variables, softmax, crossentropy

18.3.4 Using loss scaling with mixed precision

This section explains why small float16 gradients can prevent learning during mixed-precision training and how loss scaling helps avoid that problem.

• Small float16 gradient underflow: Explain why mixed-precision training can fail when gradients become very small.
  Why it matters: This explains the training problem that loss scaling addresses.
• Loss scaling optimizer options: Explain how scaling the loss helps preserve small gradients, and distinguish a fixed scaling factor from automatic scaling in Keras.
  Why it matters: It connects the numerical problem to a practical Keras remedy and its usual choice of optimizer configuration.

Key terms: mixed precision, float16, gradients, loss scaling, loss_scale_factor, LossScaleOptimizer

18.3.5 Beyond mixed precision: float8 training

This section explains why float8 training needs special handling beyond ordinary mixed precision and when its costs can outweigh its speed benefits.

• Float8 precision limits: Explain why float8 cannot replace float16 by simply changing the compute dtype in a mixed-precision forward pass.
  Why it matters: It explains why moving below float16 is not a simple configuration change.
• Keras float8 handling: Describe the scope of Keras's built-in float8 training and how it handles limited representable values.
  Why it matters: The limited layer support and extra rescaling explain what makes float8 training specialized.
• Float8 performance tradeoff: Explain why float8 training may slow a model down and identify the circumstances in which the section says it is viable.
  Why it matters: Lower precision does not guarantee faster training when its supporting operations add overhead.

Key terms: mixed precision training, float16, float8, compute_dtype, gradient updates, Keras, float8 training, Dense, EinsumDense, Embedding

18.3.6 Faster inference with quantization

This section explains how int8 quantization speeds up inference by scaling an already trained model’s values for lower-precision computation, then restoring the output’s scale.

• Int8 quantization purpose: Explain that int8 quantization converts an already trained float32 model to lower-precision weights for faster inference while aiming to preserve the numerical correctness of its forward pass.
  Why it matters: It explains why you might choose quantization after training a model.
• Scaled int8 matrix multiplication: Trace how inputs are scaled into the int8 range, cast for matrix multiplication, and then cast back and unscaled. Explain why scaling can be canceled but rounding during the int8 cast can still cause error.
  Why it matters: It shows why faster integer computation can produce a result close to the original float32 calculation.
• Built-in Keras quantization support: State that int8 quantization is built into Keras Dense, EinsumDense, and Embedding layers, enabling int8 inference for Transformer-based models that use them, rather than requiring hand implementation.
  Why it matters: The practical route is to use Keras’s layer support rather than implement the scaling math yourself.

Key terms: int8 quantization, float32, inference, matmul, abs-max scaling, int8, input scaling factors, rounding, Dense, EinsumDense

18.4 Summary