Batch size
Batch size is the number of items grouped together for processing or computation as a single batch.
Deep Learning with PyTorch, Second Edition
The number of training examples processed together in one update step.
- How it is used
- Sets how many examples a model processes together and how many input–target pairs a training-data helper selects.
- How deep it goes
- Demonstrates tensor shapes, batch selection, and a memory-limit exercise without prescribing an optimal training batch size.
- Examples
- Uses ten one-feature inputs, three padded input–target pairs, and a single image expanded for convolution.
- What you can do after
- Operate/configure it by setting a training-data batch size and checking DataLoader batch-size and worker combinations against available memory.
Read how Deep Learning with PyTorch, Second Edition explains batch size →
Natural Language Processing in Action, Second Edition
Batch size is the number of examples processed together in one training or evaluation step.
- How it is used
- Controls how many examples are processed together when training or evaluating language models.
- How deep it goes
- Shows tensor batchification and concrete configuration choices, but does not derive an optimal batch size.
- Examples
- Uses eight-token RNN batches, BERT training and evaluation batches of 24, and a GPT-2 per-device training batch of five.
- What you can do after
- Operate/configure it by setting training and evaluation batch sizes, then adjusting them for sequence length and GPU memory.
Read how Natural Language Processing in Action, Second Edition explains batch size →
Build a Large Language Model (From Scratch)
The number of training examples processed together in one update or evaluation step.
- How it is used
- Organizes tokenized examples into PyTorch DataLoader batches for GPT training, classification fine-tuning, validation, and testing.
- How deep it goes
- Explains tensor shapes and DataLoader options with runnable configurations, but does not analyze optimization or hardware-performance trade-offs.
- Examples
- Uses batch sizes of 1 and 8, shifted GPT input-target tensors, and classification batches shaped [8, 120] with labels shaped [8].
- What you can do after
- Operate/configure PyTorch DataLoader instances with batch_size, shuffle, drop_last, and worker settings, then verify returned tensor dimensions.
Read how Build a Large Language Model (From Scratch) explains batch size →
Deep Learning with Python, Third Edition
The number of sequence examples processed together in one forward or training step.
- How it is used
- Sets how many samples a model processes together and uses for each training weight update.
- How deep it goes
- Defines the batch axis, traces batches through Keras training, and gives practical tuning advice without deriving gradient statistics.
- Examples
- Uses 128-sample MNIST slices and Keras training, plus 32-vector Dataset batches whose shape is `(32, 16)`.
- What you can do after
- Operate/configure it with Keras `fit(batch_size=...)` or `Dataset.batch()`, then adjust it when training stalls.
Read how Deep Learning with Python, Third Edition explains batch size →
LLMs in Production
Batch size is the number of input examples or inference requests processed together in one computation batch.
- How it is used
- Controls how training examples or inference requests are grouped to balance throughput, memory utilization, and response latency.
- How deep it goes
- Explains power-of-two choices, memory effects, dynamic batching, and latency trade-offs without detailing batching algorithms or hardware internals.
- Examples
- Uses 16-batch memory calculations, powers-of-two values, a production batch size of 20, and BentoML or Triton dynamic batching.
- What you can do after
- Operate/configure training and inference systems by selecting batch sizes and tuning dynamic batching against memory, throughput, and latency limits.
Apache Kafka in Action
A batch size is the maximum amount of record data grouped into a producer batch before transmission.
- How it is used
- Tunes producer batching to reduce network requests and improve data throughput.
- How deep it goes
- Defines the byte limit and traces partition-based buffering through broker sends, without detailing buffer allocation internals.
- Examples
- Compares performance-test results for 10 KB records at default settings and with batch.size=100000 and linger.ms=100.
- What you can do after
- Apply it via kafka-producer-perf-test.sh and producer properties to compare throughput and latency under production-like conditions.
Machine Learning Platform Engineering
A batch size is the number of training samples processed in one iteration of model training.
- How it is used
- Configures training efficiency and resource use in a production YOLO model-training pipeline.
- How deep it goes
- Defines the training-loop parameter, explains memory and performance trade-offs, and stops short of deriving optimization behavior mathematically.
- Examples
- Uses an Ultralytics YOLO train call with batch=batch and lists batch 4 as a default example, with -1 enabling AutoBatch.
- What you can do after
- Apply it via Ultralytics YOLO configuration and adjust it when memory, speed, performance, or convergence requires change.
Read how Machine Learning Platform Engineering explains batch size →
Evolutionary Deep Learning: Genetic algorithms and neural networks
Batch size is the number of input examples processed in one training update or forward pass.
- How it is used
- Uses batch size to control how many inputs enter each forward pass while tuning neural-network and GAN training.
- How deep it goes
- Defines batch size and demonstrates parameter changes experimentally, but stops short of deriving optimization theory or prescribing universal values.
- Examples
- Demonstrates batch size through a hyperparameter notebook, data_step comparisons, training-time plots, and a GAN digit-generation exercise.
- What you can do after
- Apply it via notebook parameters by changing batch size, rerunning training, and comparing fit, plots, and training time.
Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains batch size →
Kafka for Architects
Batch size is a configuration value specifying how many records or units are grouped for collective transmission or processing.
- How it is used
- Configures how Kafka producers group records and how consumers limit records delivered in a fetch.
- How deep it goes
- Explains producer limits, waiting behavior, memory implications, and throughput-latency trade-offs, but defers detailed monitoring procedures.
- Examples
- Uses continuously arriving producer records and contrasts full batches with partially filled batches delayed by linger.ms.
- What you can do after
- Configure batch.size, linger.ms, buffer.memory, and max.poll.records while evaluating latency, throughput, and memory consequences.
Grokking AI Algorithms, Second Edition
Batch size is the number of training examples processed together in one model update.
- How it is used
- Organizes token streams into manageable inputs for Transformer training and determines how much data precedes each weight update.
- How deep it goes
- Explains batching, incomplete-batch handling, and the update cycle, while stopping short of deeper optimization theory.
- Examples
- Uses 32-token batches, a six-token-batch illustration, a three-token remainder, padding, and token-slicing code.
- What you can do after
- Operate/configure training batches, process incomplete final batches, and include batch size among tunable training hyperparameters.
Read how Grokking AI Algorithms, Second Edition explains batch size →
Transformers in Action
Batch size is the number of training examples processed in one forward and backward pass before a parameter update.
- How it is used
- Configures how many training examples contribute to each gradient step during gradient descent and fine-tuning.
- How deep it goes
- Explains gradient contribution, convergence effects, and VRAM-aware configuration, but does not provide a batch-size selection procedure.
- Examples
- Uses OPUS-100 English–Spanish translation fine-tuning and an SFTConfig setting named per_device_train_batch_size.
- What you can do after
- Operate/configure fine-tuning through TRL SFTConfig while accounting for gradient accumulation and limited VRAM.
Designing Deep Learning Systems: A guide for software engineers
The number of training examples processed together in one model-training step.
- How it is used
- Positions batch size as a training configuration tuned within hyperparameter optimization.
- How deep it goes
- Defines batch size and demonstrates search-space tuning, but does not explain its training-loop internals.
- Examples
- Uses candidate values from 8 through 256 and a 36-trial batch-size and learning-rate search.
- What you can do after
- Apply batch size through automatic hyperparameter optimization and compare training evaluation results.
Read how Designing Deep Learning Systems: A guide for software engineers explains batch size →
Related concepts
- GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.
- batchingconfigures · Batching is the grouping of multiple data items, requests, or operations so they can be processed together as a unit.
- vLLMimplemented by · vLLM is an open-source library and serving engine for high-throughput, memory-efficient inference of large language models.
- DataLoaderconfigures · PyTorch DataLoader is an iterable utility that retrieves dataset samples, optionally shuffles and transforms them, and groups them into batches.
- hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
- gradient accumulationproduced by · Gradient accumulation is a training technique that combines gradients from multiple smaller batches before applying an optimizer update to model parameters.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.