liveBook · Manning

Batch size

Batch size is the number of items grouped for processing together in a single batch or computational operation.

How 14 Manning books explain it

Deep Learning with PyTorch, Second Edition

The number of training examples processed together in one update step.

How it is used
Uses batch size to group training examples and, separately, inference requests for collective processing.
How deep it goes
Works through tensor shapes and batch sampling, and describes request-queue release conditions without prescribing an optimal batch size.
Examples
Uses a ten-sample linear input, a three-pair training batch, and a queued text-generation service.
What you can do after
Apply it using PyTorch to shape batched inputs and select matching input-label pairs.

Read how Deep Learning with PyTorch, Second Edition explains batch size →

Natural Language Processing in Action, Second Edition

Batch size is the number of examples processed together in one training or evaluation step.

How it is used
Uses batch size to organize simultaneous inputs for language-model training and classifier training or evaluation.
How deep it goes
Explains tensor batching and memory tradeoffs, then demonstrates batch parameters in fine-tuning configurations.
Examples
Uses WikiText-2 token tensors, a toxic-comment BERT classifier, and GPT-2 fine-tuning to illustrate batching.
What you can do after
Operate and configure training and evaluation batch sizes while checking GPU memory constraints.

Read how Natural Language Processing in Action, Second Edition explains batch size →

Build a Large Language Model (From Scratch)

The number of training examples processed together in one update or evaluation step.

How it is used
Organizes tokenized examples into PyTorch DataLoader batches for GPT training, classification fine-tuning, validation, and testing.
How deep it goes
Explains tensor shapes and DataLoader options with runnable configurations, but does not analyze optimization or hardware-performance trade-offs.
Examples
Uses batch sizes of 1 and 8, shifted GPT input-target tensors, and classification batches shaped [8, 120] with labels shaped [8].
What you can do after
Operate/configure PyTorch DataLoader instances with batch_size, shuffle, drop_last, and worker settings, then verify returned tensor dimensions.

Read how Build a Large Language Model (From Scratch) explains batch size →

Deep Learning with Python, Third Edition

The number of sequence examples processed together in one forward or training step.

How it is used
Controls how many samples a model processes together and uses for each training update.
How deep it goes
Explains batch axes, training updates, common sizes, dataset batching, and tuning when training stalls.
Examples
Uses MNIST training with 128 samples per batch and a TensorFlow dataset of 16-element vectors batched in groups of 32.
What you can do after
Apply it via Keras and TensorFlow to configure training batches and inspect their shapes.

Read how Deep Learning with Python, Third Edition explains batch size →

LLMs in Production

Batch size is the number of input examples or inference requests processed together in one computation batch.

How it is used
Controls how training examples or inference requests are grouped to balance throughput, memory utilization, and response latency.
How deep it goes
Explains power-of-two choices, memory effects, dynamic batching, and latency trade-offs without detailing batching algorithms or hardware internals.
Examples
Uses 16-batch memory calculations, powers-of-two values, a production batch size of 20, and BentoML or Triton dynamic batching.
What you can do after
Operate/configure training and inference systems by selecting batch sizes and tuning dynamic batching against memory, throughput, and latency limits.

Read how LLMs in Production explains batch size →

Apache Kafka in Action

A batch size is the maximum amount of record data grouped into a producer batch before transmission.

How it is used
Tunes producer batching to reduce network requests and improve data throughput.
How deep it goes
Defines the byte limit and traces partition-based buffering through broker sends, without detailing buffer allocation internals.
Examples
Compares performance-test results for 10 KB records at default settings and with batch.size=100000 and linger.ms=100.
What you can do after
Apply it via kafka-producer-perf-test.sh and producer properties to compare throughput and latency under production-like conditions.

Read how Apache Kafka in Action explains batch size →

Machine Learning Platform Engineering

A batch size is the number of training samples processed in one iteration of model training.

How it is used
Configures training efficiency and resource use in a production YOLO model-training pipeline.
How deep it goes
Defines the training-loop parameter, explains memory and performance trade-offs, and stops short of deriving optimization behavior mathematically.
Examples
Uses an Ultralytics YOLO train call with batch=batch and lists batch 4 as a default example, with -1 enabling AutoBatch.
What you can do after
Apply it via Ultralytics YOLO configuration and adjust it when memory, speed, performance, or convergence requires change.

Read how Machine Learning Platform Engineering explains batch size →

Build High Value Software Systems

The number of work items grouped together for processing or delivery as one batch.

How it is used
Uses batch size to explain how development work moves through review, feedback, and delivery.
How deep it goes
Explains effects on review, feedback, rework, and cycle time through pull-request comparisons and flaky-test workflows.
Examples
Uses contrasting shipping-rate pull requests and a flaky-test workflow that turns small changes into larger commits.
What you can do after
Apply it by splitting changes into smaller commits or pull requests to shorten feedback loops and reduce rework.

Read how Build High Value Software Systems explains batch size →

Evolutionary Deep Learning: Genetic algorithms and neural networks

Batch size is the number of input examples processed in one training update or forward pass.

How it is used
Uses batch size to control how many inputs enter each forward pass while tuning neural-network and GAN training.
How deep it goes
Defines batch size and demonstrates parameter changes experimentally, but stops short of deriving optimization theory or prescribing universal values.
Examples
Demonstrates batch size through a hyperparameter notebook, data_step comparisons, training-time plots, and a GAN digit-generation exercise.
What you can do after
Apply it via notebook parameters by changing batch size, rerunning training, and comparing fit, plots, and training time.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains batch size →

Kafka for Architects

Batch size is a configuration value specifying how many records or units are grouped for collective transmission or processing.

How it is used
Configures how Kafka producers group records and how consumers limit records delivered in a fetch.
How deep it goes
Explains producer limits, waiting behavior, memory implications, and throughput-latency trade-offs, but defers detailed monitoring procedures.
Examples
Uses continuously arriving producer records and contrasts full batches with partially filled batches delayed by linger.ms.
What you can do after
Configure batch.size, linger.ms, buffer.memory, and max.poll.records while evaluating latency, throughput, and memory consequences.

Read how Kafka for Architects explains batch size →

Grokking AI Algorithms, Second Edition

Batch size is the number of training examples processed together in one model update.

How it is used
Organizes token streams into manageable inputs for Transformer training and determines how much data precedes each weight update.
How deep it goes
Explains batching, incomplete-batch handling, and the update cycle, while stopping short of deeper optimization theory.
Examples
Uses 32-token batches, a six-token-batch illustration, a three-token remainder, padding, and token-slicing code.
What you can do after
Operate/configure training batches, process incomplete final batches, and include batch size among tunable training hyperparameters.

Read how Grokking AI Algorithms, Second Edition explains batch size →

Local Image Generation

Batch size is the number of images processed simultaneously in one generation iteration.

How it is used
Configures how many images enter generation or training simultaneously.
How deep it goes
Explains parallelism, sequential comparison, VRAM effects, and practical settings without detailing sampling internals.
Examples
Uses 30-image generation combinations, ComfyUI node configuration, and VRAM-based fine-tuning settings.
What you can do after
Operate/configure batch size for image generation, ComfyUI workflows, and fine-tuning within available VRAM.

Read how Local Image Generation explains batch size →

Transformers in Action

Batch size is the number of training examples processed in one forward and backward pass before a parameter update.

How it is used
Configures how many training examples contribute to each gradient step during gradient descent and fine-tuning.
How deep it goes
Explains gradient contribution, convergence effects, and VRAM-aware configuration, but does not provide a batch-size selection procedure.
Examples
Uses OPUS-100 English–Spanish translation fine-tuning and an SFTConfig setting named per_device_train_batch_size.
What you can do after
Operate/configure fine-tuning through TRL SFTConfig while accounting for gradient accumulation and limited VRAM.

Read how Transformers in Action explains batch size →

Designing Deep Learning Systems: A guide for software engineers

The number of training examples processed together in one model-training step.

How it is used
Positions batch size as a training configuration tuned within hyperparameter optimization.
How deep it goes
Defines batch size and demonstrates search-space tuning, but does not explain its training-loop internals.
Examples
Uses candidate values from 8 through 256 and a 36-trial batch-size and learning-rate search.
What you can do after
Apply batch size through automatic hyperparameter optimization and compare training evaluation results.

Read how Designing Deep Learning Systems: A guide for software engineers explains batch size →

Also covered in

What to know first

Related concepts

Questions about batch size

What is batch size?

Batch size is the number of items grouped for processing together in a single batch or computational operation.

Which Manning books explain batch size?

Deep Learning with PyTorch, Second Edition, Natural Language Processing in Action, Second Edition, Build a Large Language Model (From Scratch), Deep Learning with Python, Third Edition, LLMs in Production, Apache Kafka in Action, Machine Learning Platform Engineering, Build High Value Software Systems and 6 more.

What should I know before learning batch size?

Manning books that teach batch size usually introduce language model, accuracy, objective function and parameter first.

What is batch size a kind of?

Batch size is a kind of hyperparameter. A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.