Chapter 7: A deep dive on Keras

Introduction

The introduction previews advanced Keras workflows for applications that go beyond Sequential models and the default fit() loop.

7.1 A spectrum of workflows

Explains how Keras supports a range of increasingly flexible workflows without requiring users to abandon familiar tools as their needs grow.

• Progressive workflow flexibility: Explain how Keras makes simple workflows approachable while offering increasingly flexible ways to build and train models as needs become more complex.
  Why it matters: You can choose how much control you need now and take on more complexity gradually.
• Shared API interoperability: Explain why Keras components from different workflows can be combined and why earlier learning remains useful in more advanced workflows.
  Why it matters: The Keras basics you learn remain useful when you adopt more customized workflows.

Key terms: progressive disclosure of complexity, spectrum of workflows, usability, flexibility, fit(), shared APIs, Layer, Model, workflows

7.2 Different ways to build Keras models

Introduces Keras’s three model-building APIs by comparing their simplicity, flexibility, and tradeoffs.

• Keras model-building API tradeoffs: Distinguish the Sequential model, Functional API, and model subclassing by the kinds of architectures or control they offer and the tradeoffs described here.
  Why it matters: These tradeoffs help you choose an appropriate way to build a Keras model.

Key terms: Sequential model, Functional API, model subclassing

7.2.1 The Sequential model

This section shows how to build a Sequential model incrementally and how input shape determines when its weights and summary become available.

• Incremental Sequential model construction: Describe how to build a Sequential model one layer at a time with add().
  Why it matters: Adding layers one at a time makes it possible to inspect the model as it grows.
• Input-dependent weight creation: Explain why a new Sequential model may have no weights and how it becomes built.
  Why it matters: It explains why defining a model is not always enough to inspect its weights.
• Early model summary inspection: Explain how declaring the input shape in advance enables summary() during incremental construction, and what that helps you inspect.
  Why it matters: Checking output shapes as layers are added helps catch model-construction problems.

Key terms: Sequential model, add() method, weights, input shape, built, build(), Input class, summary(), output shape

7.2.2 The Functional API

This section explains how the Keras Functional API builds models as graphs of symbolic tensors, supports multiple inputs and outputs, and exposes layer connections for visualization and feature reuse.

• Functional API model topology: Explain why the Functional API is useful when a model has multiple inputs, multiple outputs, or a nonlinear arrangement of layers.
  Why it matters: It helps you choose a model-building approach that can express the connections your task requires.
• Symbolic tensor model construction: Trace how a simple Functional model is defined using an Input, layers called on symbolic tensors, and a Model constructed from the resulting inputs and outputs.
  Why it matters: Symbolic tensors let you define a model’s connections before supplying actual data.
• Multiple input training data: Describe how to supply training data to a multi-input, multi-output Functional model using ordered lists or named dictionaries.
  Why it matters: Correctly matching data to each input and output prevents confusion when training a branched model.
• Functional graph visualization: Explain what plotting a Functional model reveals and what additional information can be displayed.
  Why it matters: A plot makes the model’s connections visible and can help locate shape or layer-identity mistakes.
• Intermediate feature reuse: Explain how access to a Functional model’s layer connections enables a new model to reuse intermediate features instead of rebuilding the original model from scratch.
  Why it matters: Reusable intermediate outputs let you extend a model without recreating its earlier feature-producing layers.

Key terms: Sequential model, Functional API, graph, nonlinear topology, Input, symbolic tensor, shape, dtype, Model constructor, fit()

7.2.3 Subclassing the Model class

This section shows how to build a Keras model by subclassing Model and explains the flexibility and limitations of that approach.

• Subclassed model construction: Describe how to define and instantiate a Model subclass, including when its weights are created.
  Why it matters: This is the basic pattern for putting a model's forward pass in Python code.
• Models versus layers: Distinguish a Model subclass from a Layer subclass by their roles and available methods.
  Why it matters: The distinction explains why similar subclassing code produces objects with different jobs.
• Subclassing flexibility: Explain why Model subclassing can express models that the graph-based building pattern cannot.
  Why it matters: This explains when the extra freedom of subclassing is useful.
• Subclassing costs and visibility: Explain the debugging and inspection costs of using a subclassed model instead of a Functional model.
  Why it matters: The same freedom that enables custom behavior makes a subclassed model harder to debug and inspect.

Key terms: Model subclassing, __init__(), call(), forward pass, weights, Layer subclass, Model subclass, fit(), evaluate(), predict()

7.2.4 Mixing and matching different components

Keras model-building patterns can be combined rather than used as mutually exclusive choices.

• Interoperable Keras model patterns: Explain that choosing a Sequential model, the Functional API, or Model subclassing does not prevent using the others, and describe how Functional and subclassed components can contain one another.
  Why it matters: You can choose a Keras workflow without giving up components built with another pattern.

Key terms: Sequential model, Functional API, Model subclassing, spectrum of workflows

7.2.5 Remember: Use the right tool for the job

Explains how to choose among Keras model-building workflows, favoring the Functional API when a model can be expressed as a graph of layers.

• Functional API selection: Justify choosing the Functional API over model subclassing when a model can be expressed as a directed acyclic graph of layers. Explain its tradeoff between ease of use and flexibility and the value of direct access to layer connectivity.
  Why it matters: This gives you a practical rule for choosing how to build a Keras model.
• Functional models with subclassed layers: Explain why a Functional model can include subclassed layers and what the section says this combination achieves.
  Why it matters: You need not give up the Functional API’s advantages to use subclassed layers.

Key terms: Functional API, model subclassing, directed acyclic graph of layers, layer connectivity, Functional models, subclassed layers

7.3 Using built-in training and evaluation loops

Introduces Keras’s range of model-training workflows and two ways to customize its built-in workflow.

• Training workflow flexibility: Explain that Keras offers progressively more flexible ways to train a model, from calling fit() on data to writing a training algorithm from scratch.
  Why it matters: You can choose how much control you need without starting with the most complex approach.
• Built-in workflow customization: Describe the two customization options named here for the compile(), fit(), evaluate(), predict() workflow: providing custom metrics and passing callbacks to fit() to schedule actions during training.
  Why it matters: These options add customization while retaining the familiar built-in workflow.

Key terms: progressive disclosure of complexity, spectrum of workflows, fit(), compile(), evaluate(), predict(), custom metrics, callbacks

7.3.1 Writing your own metrics

This section explains when to write a custom Keras metric and how its state, value, and reset behavior are defined.

• Metric purpose and choice: Explain that metrics measure model performance, including differences between training and test performance, and that built-in Keras metrics usually suffice for classification and regression while unusual needs may call for a custom metric.
  Why it matters: It helps you decide whether an existing metric is enough for what you want to measure.
• Custom metric state lifecycle: Describe how a custom metric subclasses keras.metrics.Metric, updates its Keras-variable state in update_state(), returns its current value in result(), and clears its state in reset_state() so the object can be reused.
  Why it matters: These methods let Keras report a custom measurement without carrying stale state between runs.

Key terms: metrics, keras.metrics, training data, test data, keras.metrics.Metric, Keras variables, update_state(), result(), reset_state(), backpropagation

7.3.2 Using callbacks

This section explains how Keras callbacks let a training run respond to its progress, focusing on early stopping and model checkpointing.

• Callbacks during model training: Explain what a Keras callback is, how it is supplied to fit(), and why it makes training more responsive than a fixed run.
  Why it matters: Callbacks let training react to what is happening instead of simply running for a preset number of epochs.
• Validation guided early stopping: Explain how EarlyStopping decides when to interrupt training and why this avoids a wasteful retraining run.
  Why it matters: You need not guess the best training duration in advance or repeat a run just to use a better epoch count.
• Model checkpoint saving: Describe what ModelCheckpoint saves during training and how it complements EarlyStopping.
  Why it matters: The best model may occur before training stops, so saving it during the run preserves that result.

Key terms: callback, fit(), keras.callbacks, EarlyStopping, validation loss, target metric, epochs, ModelCheckpoint, model checkpointing, best model

7.3.3 Writing your own callbacks

This section shows how to create a custom Keras callback for an action during training that built-in callbacks do not cover.

• Custom callback hooks: Explain when to write a custom callback and describe how subclassing keras.callbacks.Callback lets you implement methods called at points during training. Give the section’s example of collecting per-batch losses and plotting them after each epoch.
  Why it matters: Custom callbacks let you add training-time behavior when the built-in callbacks do not meet your needs.
• Callback method arguments: Describe the information passed to callback methods through logs and the additional index passed to epoch and batch methods.
  Why it matters: These arguments give a callback access to training results and identify the epoch or batch involved.

Key terms: keras.callbacks.Callback, custom callback, on_epoch_*, on_batch_*, logs, epoch index, batch index

7.3.4 Monitoring and visualization with TensorBoard

This section explains why detailed feedback speeds model development and how to monitor Keras training with TensorBoard.

• Experiment feedback loop: Explain how experiment results inform the next model-development idea and why richer, more frequent feedback can speed progress.
  Why it matters: Useful feedback helps you decide what to try in the next experiment.
• TensorBoard monitoring capabilities: Describe what TensorBoard lets you inspect during Keras training, including live metrics and other model visualizations.
  Why it matters: TensorBoard makes information from a running experiment easier to inspect.
• Keras TensorBoard setup: Describe the basic steps for using the TensorBoard callback with fit() and opening its interface locally or in Colab.
  Why it matters: The callback connects a Keras training run to the graphs you inspect in TensorBoard.

Key terms: experiments, loop of progress, final loss, TensorBoard, training and evaluation metrics, model architecture, histograms, embeddings, keras.callbacks.TensorBoard, fit()

7.4 Writing your own training and evaluation loops

This section explains when Keras’s built-in fit() workflow may be insufficient and recalls the basic steps of a custom training loop.

• Limits of the fit workflow: Explain why the built-in fit() workflow is usually convenient but may not provide enough flexibility, particularly for learning without explicit targets or for novel research requirements.
  Why it matters: It helps you recognize when to use fit() and when a task calls for control over training logic.
• Basic training loop steps: Trace the sequence of operations in the typical training loop recalled here and relate it to what fit() does under the hood.
  Why it matters: These steps are the starting point for understanding custom training logic.

Key terms: fit() workflow, supervised learning, targets, custom training logic, forward pass, loss, gradients, model’s weights, batches

7.4.1 Training vs. inference

This section explains two details of a Keras training loop: setting the model to training mode for the forward pass and requesting gradients only for trainable weights.

• Training and inference behavior: Explain why a training-loop forward pass should call a Keras model with training=True, using Dropout as the example of a layer whose behavior differs during training and inference.
  Why it matters: Without the training flag, the forward pass may use inference behavior instead of the behavior intended for training.
• Trainable and non-trainable weights: Distinguish trainable weights from non-trainable weights and explain why a training loop retrieves gradients for model.trainable_weights rather than model.weights.
  Why it matters: The distinction keeps gradient-based updates focused on weights meant to minimize the loss.

Key terms: training Boolean argument, forward pass, Dropout layer, inference, model.trainable_weights, model.weights, trainable weights, non-trainable weights, backpropagation

7.4.2 Writing custom training step functions

This section shows how to write a supervised-learning training step with Keras on TensorFlow, PyTorch, and JAX, emphasizing how each backend obtains gradients and updates state.

• Backend-specific gradient computation: Explain why a custom training step cannot use one backend-independent gradient call, and describe TensorFlow’s use of GradientTape around the forward pass.
  Why it matters: It explains why translating a custom training step between backends requires more than changing an import.
• PyTorch backward and gradient clearing: Trace how PyTorch’s backward pass populates parameter gradients, and explain why a custom training step must clear them between steps.
  Why it matters: Forgetting to clear gradients changes what each PyTorch training step computes.
• JAX stateless loss and gradients: Explain how JAX’s stateless approach shapes the loss function, including the role of stateless_call() and has_aux in retrieving gradients alongside non-trainable-weight updates.
  Why it matters: It shows why a JAX training step must explicitly carry updated model state as well as the loss.
• Stateless optimizer state updates: Describe why a JAX training step uses the optimizer’s stateless_apply() method rather than treating every optimizer as a simple weight update.
  Why it matters: The optimizer’s changing state also has to be carried explicitly to complete a stateless JAX training step.

Key terms: GradientTape, forward pass, backend, computation graph, loss.backward(), weight.value.grad, model.zero_grad(), stateless_call(), compute_loss_and_updates(), jax.value_and_grad

7.4.3 Low-level usage of metrics

This subsection shows how to track and reset Keras metrics in a low-level training loop, with a caveat for JAX stateless functions.

• Metrics in low-level loops: Describe how to update a Keras metric for each batch, read its current value, track an average scalar loss, and reset results between epochs or evaluations.
  Why it matters: These calls let you report meaningful metrics without carrying results over from a previous epoch or evaluation.
• JAX stateless metric constraint: Explain why state-modifying metric methods cannot be used inside a JAX stateless function and identify the stateless metrics API as the alternative named here.
  Why it matters: The usual metric calls need a different API when metric handling takes place inside a JAX stateless function.

Key terms: update_state(), result(), keras.metrics.Mean, reset_state(), JAX, stateless function, stateless metrics API

7.4.4 Using fit() with a custom training loop

This section shows how to run a custom training step through Keras fit() instead of writing an entire training loop.

• Custom training middle ground: Explain why overriding a training step is a middle ground between ordinary fit() and a training loop written from scratch.
  Why it matters: You can choose custom learning logic without taking responsibility for the whole training loop.
• Custom train_step integration: Describe how to make fit() use a custom training step, including what train_step() returns and how training is started.
  Why it matters: This is the connection that lets your per-batch learning logic run inside fit().
• JAX loss reporting caveat: Distinguish the loss reported in this section's simplified JAX example from the loss reporting used for the other two backends.
  Why it matters: It keeps the example’s reported loss from being mistaken for a moving average.

Key terms: fit(), custom training step function, training loop, keras.Model, train_step(), compile(), metric names, loss, JAX, per-batch loss

7.4.5 Handling metrics in a custom train_step()

This section identifies how a custom train_step() accesses the loss and metrics configured with compile().

• Compiled loss and metric access: Explain how a custom train_step() can access the loss and metrics supplied to compile(), including what each access point contains.
  Why it matters: These access points let a custom training step use the loss and metrics configured through compile().

Key terms: compile(), self.compute_loss, regularization losses, self.metrics

7.5 Summary