What does section 3.1, The world as floating-point numbers, cover?

The section explains why neural networks represent data as floating-point numbers and introduces PyTorch tensors as the structures used to hold those numbers. Explain why real-world data must be encoded as floating-point input for a network and its output decoded into something usable. Describe how a deep network transforms data in stages and what its intermediate representations express. Describe a deep-learning tensor as a multidimensional array and explain how its indices, dimensions, and shape relate.

What does section 3.2, Tensors: Multidimensional arrays, cover?

Defines a PyTorch tensor as an array of numbers whose elements can be accessed by index, including with multiple indices. Explain that a tensor is PyTorch’s fundamental data structure: an array holding numbers that can be accessed individually by index, including with multiple indices.

What does section 3.2.1, From Python lists to PyTorch tensors, cover?

The section uses a Python list to introduce zero-based indexing and explains why tensors are useful for representing and manipulating data. Explain why the section moves from Python lists to tensors: tensors can represent many kinds of data, and operations on them support expressive, efficient slicing and manipulation. Describe how the section accesses the first element of a Python list of numbers, and relate that example to its comparison with tensor indexing.

What does section 3.2.2, Constructing our first tensors, cover?

The section constructs a simple PyTorch tensor and shows that its elements can be accessed and changed. Describe how the example creates a one-dimensional PyTorch tensor containing three 1.0 values, then accesses an element by its zero-based index and assigns it a new value.

What does section 3.2.3, The essence of tensors, cover?

This section explains how tensors store homogeneous numeric values, organize coordinates by dimensions, and return a view when a row is selected. Distinguish how Python lists and tensors store numbers, including the memory implication of storing 32-bit floats in a tensor. Describe how a 2D tensor organizes the triangle's coordinates and how its shape and indices identify points and individual coordinates. Explain what selecting the first point from the 2D points tensor returns and whether its values are copied.

What does section 3.3, Indexing tensors, cover?

This section shows that PyTorch tensors use Python-style range indexing to select elements, including along each dimension. Explain how Python-style range indexing can select part of a PyTorch tensor, such as all points except the first, and how ranges can be used for each dimension.

What does section 3.4, Broadcasting, cover?

Explains how broadcasting lets PyTorch perform element-wise operations on tensors with compatible but different shapes. Explain how to determine whether tensor shapes can be broadcast: compare dimensions from right to left, right-aligning shapes of different lengths, and require each compared pair to be equal or include a 1. Describe how a size-1 dimension can be thought of as virtually duplicated so an operation is applied element-wise, and use the section’s examples to predict the resulting shape.

What does section 3.5, Named tensors, cover?

This section uses an image-conversion example to show why positional tensor dimensions can cause mistakes and how named tensors can make dimension alignment clearer. Explain why code that refers to dimensions by position can be error-prone, using the section’s single-image and batched-image examples. Describe how named tensors make axis meanings explicit and how this section assigns or changes dimension names. Explain how names affect tensor operations, including explicit alignment and returning to unnamed tensors when needed.

What does section 3.6, Tensor element types, cover?

This section explains why Python numbers and lists are inefficient for large-scale numerical work and why tensors track a single numeric type for their elements. Explain why standard Python numbers and lists can be inefficient for large collections of numerical data: numbers are boxed objects, lists hold pointers and lack efficient numerical operations, and Python-level computation is slower than optimized compiled code. Explain that tensors support efficient low-level numerical storage and operations behind a high-level API, which requires their elements to be numbers of the same type and PyTorch to track that type.

What does section 3.6.1, Specifying the numeric type with dtype, cover?

This section explains how the dtype argument sets a tensor’s numeric type and identifies PyTorch’s default and available types. Explain what dtype specifies when constructing a tensor and state the default dtype. Describe the kinds of dtypes listed and distinguish bfloat16 from float16 at the level explained here.

What does section 3.6.2, A dtype for every occasion, cover?

This section explains why float32 and int64 are common tensor dtypes and how tensor comparisons produce bool tensors. Explain why neural-network computations typically use float32 rather than float64, and when float16 may be useful. State which dtype PyTorch expects for tensors used as indices and what dtype results by default when creating a tensor from integer arguments. Describe the dtype and meaning of the tensor produced by a comparison such as points > 1.0.

What does section 3.6.3, Managing a tensor’s dtype attribute, cover?

This section shows how to set, inspect, and change a tensor’s dtype, and explains what happens when operations mix input types. Describe how to specify a tensor’s numeric type when constructing it, inspect its dtype attribute, and cast a tensor using a dtype-named method or to. Explain what happens when an operation mixes input types and what that implies for keeping computation at 32 bits.

What should I know before reading chapter 3?

Ready, set, almost run (chapter 2): The image tensor and added batch dimension used in the example.

Chapter 3: It starts with a tensor

What you will learn in chapter 3 of Deep Learning with PyTorch, Second Edition

The introduction frames deep learning as learning transformations between data representations and explains why PyTorch begins with numeric inputs and tensors.

After this chapter you will be able to

Learned data transformations. Explain that a deep learning system transforms inputs such as images or text into outputs such as labels, numbers, or other images or text by finding input-to-output patterns across many examples.
Numeric inputs and tensors. Describe why working with floating-point numbers and tensors is a starting point for using PyTorch in deep learning.
Numeric input and output. Explain why real-world data must be encoded as floating-point input for a network and its output decoded into something usable.
Learned intermediate representations. Describe how a deep network transforms data in stages and what its intermediate representations express.
Tensor dimensions and shape. Describe a deep-learning tensor as a multidimensional array and explain how its indices, dimensions, and shape relate.
Tensor array structure. Explain that a tensor is PyTorch’s fundamental data structure: an array holding numbers that can be accessed individually by index, including with multiple indices.
Tensor representation and manipulation. Explain why the section moves from Python lists to tensors: tensors can represent many kinds of data, and operations on them support expressive, efficient slicing and manipulation.
Creating and changing a tensor. Describe how the example creates a one-dimensional PyTorch tensor containing three 1.0 values, then accesses an element by its zero-based index and assigns it a new value.
Homogeneous tensor storage. Distinguish how Python lists and tensors store numbers, including the memory implication of storing 32-bit floats in a tensor.
Point tensor shape and indexing. Describe how a 2D tensor organizes the triangle's coordinates and how its shape and indices identify points and individual coordinates.
Selected row as tensor view. Explain what selecting the first point from the 2D points tensor returns and whether its values are copied.
Tensor range indexing. Explain how Python-style range indexing can select part of a PyTorch tensor, such as all points except the first, and how ranges can be used for each dimension.
Broadcasting shape compatibility. Explain how to determine whether tensor shapes can be broadcast: compare dimensions from right to left, right-aligning shapes of different lengths, and require each compared pair to be equal or include a 1.
Virtual expansion during operations. Describe how a size-1 dimension can be thought of as virtually duplicated so an operation is applied element-wise, and use the section’s examples to predict the resulting shape.
Positional Dimension Pitfalls. Explain why code that refers to dimensions by position can be error-prone, using the section’s single-image and batched-image examples.
Named Dimension Alignment. Explain how names affect tensor operations, including explicit alignment and returning to unnamed tensors when needed.
Uniform tensor element types. Explain that tensors support efficient low-level numerical storage and operations behind a high-level API, which requires their elements to be numbers of the same type and PyTorch to track that type.
Tensor dtype and default. Explain what dtype specifies when constructing a tensor and state the default dtype.
Floating-Point Precision Tradeoffs. Explain why neural-network computations typically use float32 rather than float64, and when float16 may be useful.
Setting and checking tensor dtype. Describe how to specify a tensor’s numeric type when constructing it, inspect its dtype attribute, and cast a tensor using a dtype-named method or to.
Mixed input type conversion. Explain what happens when an operation mixes input types and what that implies for keeping computation at 32 bits.
Tensor operation call forms. Explain that most tensor operations can be called either through the torch module or as methods on a tensor, and that the two forms are interchangeable.
Storage and tensor views. Explain that a torch.Storage instance holds a one-dimensional, contiguous block of numbers of a given type, while a tensor is a view that indexes that storage using an offset and per-dimension strides.
Shared storage across views. Describe how multiple tensors can index the same storage differently, as when points[0] shares storage with points but selects less data and has a different dimensionality. Explain why an alternate view can be created quickly.
Tensor shape and storage layout. Explain that a tensor can report three rows and two columns while its underlying storage is a contiguous, one-dimensional array of six values. Describe tensor indexing as translating a pair of indices into a storage location.
In-place tensor methods. Distinguish tensor methods with a trailing underscore from methods without one, and describe what zero_ does.
Tensor storage indexing metadata. Explain how a tensor’s size, storage offset, and stride describe its elements within storage: size gives the number of elements along each dimension, offset locates the first element, and stride gives the storage steps to the next element along each dimension.
Storage offsets and strides. Explain how a tensor’s storage offset and strides determine where its indexed elements are found in storage.
Subtensor views and shared storage. Describe what changes when indexing a point to obtain a subtensor, and why that subtensor is a view rather than a copy.
No-copy tensor transposition. Describe how transposing points into points_t rearranges its rows and columns without copying its stored elements.
Higher-Dimensional Tensor Transposition. Explain that PyTorch can transpose a multidimensional tensor by specifying the two dimensions to swap, flipping the corresponding shape and stride.
Contiguous tensor storage layout. Explain what happens when contiguous is called on a noncontiguous tensor, using the transposed points tensor as the example.
GPU tensor storage and computation. Explain that a PyTorch tensor can be transferred from CPU memory to a GPU, where PyTorch uses GPU-specific routines for its operations and can perform massively parallel computation.
Tensor device and placement. Explain that a tensor’s device specifies where its data is stored, and describe how to create a tensor on a GPU or copy an existing tensor between CPU and GPU.
GPU computation and residency. Trace what happens when an operation is performed on a GPU tensor, including where its result stays and why further operations can run there.
Tensor and NumPy array interoperability. Explain that tensors and NumPy arrays can be converted in either direction, and describe when they share memory rather than requiring a copy.
Tensor API and representation. Explain why a PyTorch tensor need not be stored as an ordinary multidimensional array, even though that is the kind used for most of the book.
Dispatch to tensor backends. Describe how PyTorch calls the appropriate computation functions for different tensor implementations, and distinguish this extensible dispatch from numeric-type switching within a backend.
Saving and reloading tensors. Explain why a valuable tensor might be saved to a file and loaded later, and describe the PyTorch options mentioned for specifying that file.
Saving tensors under HDF5 keys. Describe how the section saves a tensor as an HDF5 dataset and what the dataset key represents.
Selective loading and file lifetime. Trace how h5py loads selected stored data into a tensor and why the file must remain open until that access is complete.

Cover of Deep Learning with PyTorch, Second Edition

liveBook manning.com

Read chapter 3 on liveBook Get the book at manning.com

What it builds on

Ready, set, almost run (chapter 2) · The image tensor and added batch dimension used in the example.

Section by section

Introduction

The introduction frames deep learning as learning transformations between data representations and explains why PyTorch begins with numeric inputs and tensors.

Learned data transformations: Explain that a deep learning system transforms inputs such as images or text into outputs such as labels, numbers, or other images or text by finding input-to-output patterns across many examples.
Why it matters: This connects the applications from the previous chapter to a common idea of what the system learns to do.
Numeric inputs and tensors: Describe why working with floating-point numbers and tensors is a starting point for using PyTorch in deep learning.
Why it matters: It explains why the book turns from familiar data such as images to tensors.

Key terms: representation, input-to-output relationship, patterns, floating-point numbers, tensors

3.1 The world as floating-point numbers

The section explains why neural networks represent data as floating-point numbers and introduces PyTorch tensors as the structures used to hold those numbers.

Numeric input and output: Explain why real-world data must be encoded as floating-point input for a network and its output decoded into something usable.
Why it matters: This explains why preparing data as numbers is part of using a neural network.
Learned intermediate representations: Describe how a deep network transforms data in stages and what its intermediate representations express.
Why it matters: Intermediate representations connect a network’s numeric calculations to the features it learns to use.
Tensor dimensions and shape: Describe a deep-learning tensor as a multidimensional array and explain how its indices, dimensions, and shape relate.
Why it matters: Dimensions and shape provide the vocabulary for describing the data PyTorch handles.
PyTorch tensor capabilities: Distinguish the stated capabilities of PyTorch tensors from NumPy arrays while noting their interoperability.
Why it matters: These capabilities help explain why the book uses PyTorch tensors rather than treating them as interchangeable with NumPy arrays.

Key terms: floating-point numbers, encode, decode, intermediate representations, weights, tensor, multidimensional array, dimension, shape, index

3.2 Tensors: Multidimensional arrays

Defines a PyTorch tensor as an array of numbers whose elements can be accessed by index, including with multiple indices.

Tensor array structure: Explain that a tensor is PyTorch’s fundamental data structure: an array holding numbers that can be accessed individually by index, including with multiple indices.
Why it matters: This gives you the basic picture of what PyTorch tensors hold and how their elements are accessed.

Key terms: tensor, array, index, multiple indices

3.2.1 From Python lists to PyTorch tensors

The section uses a Python list to introduce zero-based indexing and explains why tensors are useful for representing and manipulating data.

Tensor representation and manipulation: Explain why the section moves from Python lists to tensors: tensors can represent many kinds of data, and operations on them support expressive, efficient slicing and manipulation.
Why it matters: This is the reason to use tensors rather than rely only on Python lists for numerical data.
Zero-based list indexing: Describe how the section accesses the first element of a Python list of numbers, and relate that example to its comparison with tensor indexing.
Why it matters: The familiar list example gives you a starting point for discussing tensor indexing.

Key terms: tensor, slice, Python lists, list indexing, zero-based index

3.2.2 Constructing our first tensors

The section constructs a simple PyTorch tensor and shows that its elements can be accessed and changed.

Creating and changing a tensor: Describe how the example creates a one-dimensional PyTorch tensor containing three 1.0 values, then accesses an element by its zero-based index and assigns it a new value.
Why it matters: This small example establishes how to create a tensor and work with its elements.

Key terms: torch module, one-dimensional tensor, zero-based index

3.2.3 The essence of tensors

This section explains how tensors store homogeneous numeric values, organize coordinates by dimensions, and return a view when a row is selected.

Homogeneous tensor storage: Distinguish how Python lists and tensors store numbers, including the memory implication of storing 32-bit floats in a tensor.
Why it matters: This distinction explains why a tensor is more than a Python list with a different interface.
Point tensor shape and indexing: Describe how a 2D tensor organizes the triangle's coordinates and how its shape and indices identify points and individual coordinates.
Why it matters: Thinking in dimensions makes the point data easier to inspect and index.
Selected row as tensor view: Explain what selecting the first point from the 2D points tensor returns and whether its values are copied.
Why it matters: A selected row can be another tensor without requiring a copy of all its values.

Key terms: Python objects, unboxed C numeric types, homogeneous, metadata, 2D tensor, shape, dimension, index, view, underlying data

3.3 Indexing tensors

This section shows that PyTorch tensors use Python-style range indexing to select elements, including along each dimension.

Tensor range indexing: Explain how Python-style range indexing can select part of a PyTorch tensor, such as all points except the first, and how ranges can be used for each dimension.
Why it matters: Range indexing lets you select the part of a tensor you need, even when it has multiple dimensions.

Key terms: range indexing notation, tensor’s dimensions

3.4 Broadcasting

Explains how broadcasting lets PyTorch perform element-wise operations on tensors with compatible but different shapes.

Broadcasting shape compatibility: Explain how to determine whether tensor shapes can be broadcast: compare dimensions from right to left, right-aligning shapes of different lengths, and require each compared pair to be equal or include a 1.
Why it matters: Checking dimensions from the right helps you predict whether a tensor operation will work.
Virtual expansion during operations: Describe how a size-1 dimension can be thought of as virtually duplicated so an operation is applied element-wise, and use the section’s examples to predict the resulting shape.
Why it matters: Virtual duplication explains how differently shaped tensors can participate in one element-wise operation.

Key terms: broadcasting, compatible dimensions, right-align, element-wise, virtually duplicated, scalar

3.5 Named tensors

This section uses an image-conversion example to show why positional tensor dimensions can cause mistakes and how named tensors can make dimension alignment clearer.

Positional Dimension Pitfalls: Explain why code that refers to dimensions by position can be error-prone, using the section’s single-image and batched-image examples.
Why it matters: Recognizing what each axis means helps prevent image-processing code from using the wrong dimension.
Assigning Tensor Dimension Names: Describe how named tensors make axis meanings explicit and how this section assigns or changes dimension names.
Why it matters: Dimension names let code express an axis’s meaning instead of leaving readers to infer it from its position.
Named Dimension Alignment: Explain how names affect tensor operations, including explicit alignment and returning to unnamed tensors when needed.
Why it matters: Name checks can expose alignment errors, but the programmer still has to align dimensions explicitly.

Key terms: dimension, axis, channel dimension, batch dimension, named tensors, names, refine_names, rename, broadcasting, align_as

3.6 Tensor element types

This section explains why Python numbers and lists are inefficient for large-scale numerical work and why tensors track a single numeric type for their elements.

Python numerical storage limitations: Explain why standard Python numbers and lists can be inefficient for large collections of numerical data: numbers are boxed objects, lists hold pointers and lack efficient numerical operations, and Python-level computation is slower than optimized compiled code.
Why it matters: These limitations explain why numerical libraries use structures other than ordinary Python lists.
Uniform tensor element types: Explain that tensors support efficient low-level numerical storage and operations behind a high-level API, which requires their elements to be numbers of the same type and PyTorch to track that type.
Why it matters: The shared element type is part of what lets tensors serve as efficient numerical data structures.

Key terms: boxing, Python lists, Python interpreter, tensor, numeric type, high-level API

3.6.1 Specifying the numeric type with dtype

This section explains how the dtype argument sets a tensor’s numeric type and identifies PyTorch’s default and available types.

Tensor dtype and default: Explain what dtype specifies when constructing a tensor and state the default dtype.
Why it matters: Choosing a dtype determines how tensor values are represented.
Available numeric type families: Describe the kinds of dtypes listed and distinguish bfloat16 from float16 at the level explained here.
Why it matters: The available types differ in the values and precision they can represent.

Key terms: dtype, tensor constructors, torch.float32, torch.float16, torch.bfloat16, torch.uint8, torch.bool

3.6.2 A dtype for every occasion

This section explains why float32 and int64 are common tensor dtypes and how tensor comparisons produce bool tensors.

Floating-Point Precision Tradeoffs: Explain why neural-network computations typically use float32 rather than float64, and when float16 may be useful.
Why it matters: Precision choices affect memory use and computing time, not just how numbers are represented.
Integer Indexing Tensor Type: State which dtype PyTorch expects for tensors used as indices and what dtype results by default when creating a tensor from integer arguments.
Why it matters: The default dtype of a tensor made from integers matches the dtype PyTorch expects for tensor indexing.
Boolean Predicate Results: Describe the dtype and meaning of the tensor produced by a comparison such as points > 1.0.
Why it matters: A tensor comparison gives a separate true-or-false result for each element.

Key terms: float32, 64-bit floating-point precision, 16-bit floating-point, half-precision, indexing tensors, 64-bit integer, int64, predicates, bool tensors

3.6.3 Managing a tensor’s dtype attribute

This section shows how to set, inspect, and change a tensor’s dtype, and explains what happens when operations mix input types.

Setting and checking tensor dtype: Describe how to specify a tensor’s numeric type when constructing it, inspect its dtype attribute, and cast a tensor using a dtype-named method or to.
Why it matters: These options let you choose and verify the numeric type your tensor uses.
Mixed input type conversion: Explain what happens when an operation mixes input types and what that implies for keeping computation at 32 bits.
Why it matters: One higher-bit-width input can make an operation use a larger type than intended.

Key terms: dtype, constructor, casting method, to, input types, larger type, 32-bit computation

3.7 The tensor API

Introduces the two ways to call most tensor operations and shows how the online documentation organizes the tensor API.

Tensor operation call forms: Explain that most tensor operations can be called either through the torch module or as methods on a tensor, and that the two forms are interchangeable.
Why it matters: You can recognize the same operation when it appears in either form.
Tensor documentation categories: Describe how the online PyTorch documentation groups tensor operations and use those groups to identify where to look for an operation.
Why it matters: The categories give you a practical way to find operations without memorizing the entire API.

Key terms: torch module, methods of a tensor object, transpose, online docs, Creation ops, Math ops, Random sampling, Serialization

3.8 Tensors: Scenic views of storage

This section explains how tensors view numerical data in storage and why multiple views can share memory efficiently.

Storage and tensor views: Explain that a torch.Storage instance holds a one-dimensional, contiguous block of numbers of a given type, while a tensor is a view that indexes that storage using an offset and per-dimension strides.
Why it matters: This gives you a useful mental picture of how tensors work under the hood.
Shared storage across views: Describe how multiple tensors can index the same storage differently, as when points[0] shares storage with points but selects less data and has a different dimensionality. Explain why an alternate view can be created quickly.
Why it matters: Shared storage explains why another view of tensor data can be quick to create.

Key terms: torch.Storage, Tensor, view, offset, per-dimension strides, storage, tensor views, points[0]

3.8.1 Indexing into storage

This section shows how a multidimensional tensor refers to one-dimensional storage and how changes to that storage affect the tensor.

Tensor shape and storage layout: Explain that a tensor can report three rows and two columns while its underlying storage is a contiguous, one-dimensional array of six values. Describe tensor indexing as translating a pair of indices into a storage location.
Why it matters: It explains why tensor indexing and storage indexing describe the same data in different ways.
Changes through shared storage: Describe what happens to a tensor when a value in its underlying storage is changed.
Why it matters: It shows that the tensor refers to the stored values rather than holding an unaffected copy.

Key terms: tensor, storage, contiguous array, referring tensor

3.8.2 Modifying stored values: In-place operations

This section explains how a trailing underscore distinguishes in-place tensor methods from operations that return a new tensor.

In-place tensor methods: Distinguish tensor methods with a trailing underscore from methods without one, and describe what zero_ does.
Why it matters: The underscore tells you whether an operation will change the tensor you already have.

Key terms: in place, trailing underscore, zero_, source tensor

3.9 Tensor metadata: Size, offset, and stride

Introduces the size, storage offset, and stride metadata that, together with storage, define a tensor and how its elements are indexed.

Tensor storage indexing metadata: Explain how a tensor’s size, storage offset, and stride describe its elements within storage: size gives the number of elements along each dimension, offset locates the first element, and stride gives the storage steps to the next element along each dimension.
Why it matters: These three pieces of metadata explain how a tensor identifies its elements in storage.

Key terms: storage, size, shape, storage offset, stride

3.9.1 Views of another tensor’s storage

This section explains how tensor views use size, storage offset, and stride to access shared storage without copying data.

Storage offsets and strides: Explain how a tensor’s storage offset and strides determine where its indexed elements are found in storage.
Why it matters: Offsets and strides explain how a tensor can index storage without arranging its values in a new block of memory.
Subtensor views and shared storage: Describe what changes when indexing a point to obtain a subtensor, and why that subtensor is a view rather than a copy.
Why it matters: Shared storage is why extracting a subtensor can be inexpensive.
Shared mutations and cloning: Explain why changing a subtensor can change its original tensor and how cloning avoids that side effect.
Why it matters: It helps you recognize when an edit to a view will unexpectedly alter the original tensor.

Key terms: storage offset, stride, storage, subtensor, view, size, clone

3.9.2 Transposing without copying

The section shows how transposing a two-dimensional tensor changes its shape and stride while sharing the original storage.

Stride and storage positions: Explain how the stride of the original points tensor determines where its elements are found in storage.
Why it matters: Those storage steps make the effect of transposing the tensor easier to see.
No-copy tensor transposition: Describe how transposing points into points_t rearranges its rows and columns without copying its stored elements.
Why it matters: A transpose can change how you index the data without moving the data itself.

Key terms: stride, storage, points, transpose, t function, shape, points_t

3.9.3 Transposing in higher dimensions

The section extends transposition to multidimensional tensors and explains why contiguous storage can make ordered access efficient.

Higher-Dimensional Tensor Transposition: Explain that PyTorch can transpose a multidimensional tensor by specifying the two dimensions to swap, flipping the corresponding shape and stride.
Why it matters: It lets you describe a tensor transpose in terms of its dimensions and metadata, not just its values.
Contiguous Storage Access: Describe what the section calls a contiguous tensor and why visiting its values in storage order can be efficient.
Why it matters: Storage order helps explain when accessing tensor values can be faster.

Key terms: transpose, shape, stride, contiguous, storage, data locality

3.9.4 Contiguous tensors

This section explains how to make a noncontiguous tensor contiguous and why some PyTorch operations require that layout.

Contiguous tensor storage layout: Explain what happens when contiguous is called on a noncontiguous tensor, using the transposed points tensor as the example.
Why it matters: It connects a tensor’s stride to how its values are arranged in storage.
Contiguity requirements for operations: Describe why an operation may require an explicit call to contiguous and what happens when the tensor is already contiguous.
Why it matters: It explains when the explicit conversion is needed and why calling it on an already contiguous tensor is harmless.

Key terms: contiguous, transpose, stride, storage, view

3.10 Moving tensors to the GPU

The section explains that PyTorch tensors can be stored and operated on using GPUs, and describes PyTorch’s support for different accelerator hardware.

GPU tensor storage and computation: Explain that a PyTorch tensor can be transferred from CPU memory to a GPU, where PyTorch uses GPU-specific routines for its operations and can perform massively parallel computation.
Why it matters: This is the basic reason to move tensor computations from the CPU to a GPU.
Accelerator support and extensibility: Describe the accelerator platforms named in the section and explain that PyTorch’s abstractions are intended to let new hardware vendors add support without changes for existing users.
Why it matters: PyTorch’s accelerator support is not limited to one GPU vendor.

Key terms: tensor, CPU, GPU, GPU-specific routines, CUDA, ROCm, Metal Performance Shaders (MPS), torch_xla, hardware vendors

3.10.1 Managing a tensor’s device attribute

Explains how a tensor’s device controls where its data lives and how moving data to a GPU affects subsequent operations.

Tensor device and placement: Explain that a tensor’s device specifies where its data is stored, and describe how to create a tensor on a GPU or copy an existing tensor between CPU and GPU.
Why it matters: Knowing where tensor data lives helps you put computations on the intended hardware.
GPU computation and residency: Trace what happens when an operation is performed on a GPU tensor, including where its result stays and why further operations can run there.
Why it matters: Keeping successive operations on the GPU avoids assuming that every result must be moved back to the CPU.

Key terms: device, GPU, CPU, to method, cuda, GPU memory, GPU-based tensors, user-facing API

3.11 NumPy interoperability

This section explains how PyTorch tensors and NumPy arrays interoperate through shared CPU memory, with a GPU-copy exception and a numeric-type consideration.

Tensor and NumPy array interoperability: Explain that tensors and NumPy arrays can be converted in either direction, and describe when they share memory rather than requiring a copy.
Why it matters: It lets you use NumPy-based Python tools while recognizing when conversion shares data and when it copies it.
Default floating-point type difference: Distinguish PyTorch’s and NumPy’s default floating-point types and state what to check after converting to a tensor.
Why it matters: Conversion does not remove the need to check that the tensor has the numeric type you intend to use.

Key terms: NumPy array, tensor storage, underlying buffer, CPU RAM, GPU, 32-bit floating-point, 64-bit, dtype torch.float

3.12 Generalized tensors are tensors, too

The section explains that PyTorch’s tensor API can work with different storage representations and hardware through dispatching.

Tensor API and representation: Explain why a PyTorch tensor need not be stored as an ordinary multidimensional array, even though that is the kind used for most of the book.
Why it matters: It lets you recognize other tensor implementations without treating them as a different programming interface.
Dispatch to tensor backends: Describe how PyTorch calls the appropriate computation functions for different tensor implementations, and distinguish this extensible dispatch from numeric-type switching within a backend.
Why it matters: Dispatch explains how the same tensor-facing operations can work across different backends.
Dense and sparse layouts: Distinguish the usual dense or strided tensors from the sparse tensors used as an example of another data representation.
Why it matters: Sparse tensors provide a concrete example of how tensor storage can differ.

Key terms: tensor API, tensor, multidimensional arrays, dispatcher, user-facing API, backend functions, numeric types, dense, strided, sparse tensors

3.13 Serializing tensors

This section explains how to save and reload tensors with PyTorch and why its serialization format may not suit software outside PyTorch.

Saving and reloading tensors: Explain why a valuable tensor might be saved to a file and loaded later, and describe the PyTorch options mentioned for specifying that file.
Why it matters: Saving a tensor lets you reuse valuable data rather than recreate it each time a program runs.
PyTorch format interoperability: Distinguish quick PyTorch-only tensor serialization from a format readable by other software.
Why it matters: The software that needs to read a saved tensor determines whether PyTorch’s quick format is sufficient.

Key terms: serialize, pickle, storage, file descriptor, interoperable, file format

3.13.1 Serializing to HDF5 with h5py

This section shows how h5py saves tensors in interoperable HDF5 files and loads selected data back into PyTorch tensors.

HDF5 for interoperable storage: Explain when interoperable tensor storage may be useful and how HDF5 and h5py support it.
Why it matters: This explains why you might choose HDF5 rather than a PyTorch-only way to save tensors.
Saving tensors under HDF5 keys: Describe how the section saves a tensor as an HDF5 dataset and what the dataset key represents.
Why it matters: The NumPy conversion and dataset key connect a PyTorch tensor to its stored HDF5 representation.
Selective loading and file lifetime: Trace how h5py loads selected stored data into a tensor and why the file must remain open until that access is complete.
Why it matters: Selective access avoids loading unwanted data, while the file’s lifetime determines when a dataset remains usable.

Key terms: HDF5, h5py, NumPy arrays, create_dataset, coords, NumPy array, dataset, torch.from_numpy, tensor’s storage

3.14 Conclusion

The section closes the tensor groundwork and previews further tensor topics and real-world data representation.

3.15 Exercises

3.16 Summary

Where this is used later

Learning is just parameter estimation (chapter 5) · Provides the input-to-output framing for learning a model from data.
(chapter intro) (chapter 4) · The next chapter applies this need to representing different kinds of real-world data.
Composing a multilayer network (chapter 6) · Composing a multilayer network builds on the idea of successive representations.
Working with images (chapter 4) · Representing images provides a concrete example of data stored in tensors.
Finding thresholds (chapter 4) · Indexing tensors
Training, validation, and overfitting (chapter 5) · Tensor indexing allows a permutation of indices to select reordered elements.
From problem back to PyTorch (chapter 5) · Introduces broadcasting.
Changing the layout (chapter 4) · Changing image layouts requires tracking which dimension represents channels.

Concepts explained in this chapter

arrayA container for indexed collections of values.
floating-point arithmeticNumeric values represented approximately in binary floating-point form, used here as the tensor element type for loaded data.
viewA tensor representation that shares underlying storage while presenting the same data with a different shape or interpretation.
broadcastingA shape-alignment rule for element-wise operations that lets arrays or tensors of different shapes work together when dimensions are compatible.
tensor storageA one-dimensional memory buffer underlying tensor views.
transpositionAn operation that rearranges the dimensions of a tensor by swapping selected axes, changing its shape and stride without changing its values.
strideTensor metadata describing its shape and how elements are laid out and indexed in storage, including step size between positions.
data representationAn internal data encoding produced or consumed by a model.
tensor operationAn operation provided by PyTorch that acts on tensor data, either as a torch-module function or as a tensor method.
to methodThe PyTorch tensor method used to move tensors and model parameters to a specified device or change their type.
indexingSelecting tensor elements or slices by using tensor values as indices or masks.
mutationA method call that modifies an existing tensor directly rather than creating a separate result tensor.
PyTorchThe PyTorch core Python library namespace that exposes tensor operations and functions such as mathematical elementwise operations.
tensor shapeThe sizes of an array or tensor along each axis, here specifically the channels-height-width layout for image tensors.
parallelismThe practice of dividing work into tasks that execute simultaneously.
scalarA single numeric value, contrasted here with multi-dimensional feature representations.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.