liveBook · Manning

Scalability

As explained in Deep Learning with R, Third Edition

Scalability is the ability of a system or method to handle increasing amounts of data or computation without becoming impractical.

Scalability is the ability of a system or method to handle increasing amounts of data or computation without becoming impractical. In deep learning, small-batch training supports datasets of arbitrary size, while GPU and specialized-hardware parallelization makes larger workloads feasible. The limiting factor is available parallel computing power. section 1.7

Scaling also describes increasing model size, training data, or computation. These choices involve tradeoffs: a fixed budget can fund more data or a larger model, and a smaller model may be preferable when deployment hardware is expensive. More scale generally improves language-model performance, but data availability and computational cost constrain the strategy. section 16.5 section 20.2

Useful distinctions include:

  • Data scalability: process larger datasets through repeated small-batch updates rather than requiring the entire dataset at once. section 1.7
  • Compute scalability: use parallel hardware and increasing computation to train larger or more capable models. section 16.5 section 1.7
  • Practical scalability: balance performance against training cost, inference hardware, and the availability of high-quality data. section 16.5
  • Geometric scaling: multiply coordinates by a diagonal matrix to enlarge or shrink an object horizontally or vertically. section 2.3.5

Choosing what to scale

Given a fixed pretraining budget, compare spending compute on a larger model with spending it on more training data. GPT-3 is presented as an example of a model that was too large for its computing budget, making a smaller model trained on more data the potentially better choice. Deployment may likewise favor a smaller model that fits cheaper hardware. section 16.5

Figure 2.10 2D scaling as a matrix product
figure

Use the figure to see horizontal and vertical scaling represented as multiplication by a diagonal matrix. section 2.3.5

Figure 16.8 LLM parameter counts and pretraining dataset sizes over time. Many recent proprietary LLMs (e.g., GPT-4, Gemini) are not included because model details have not been disclosed.
figure

Use the figure to compare LLM parameter counts with pretraining dataset sizes across releases from 2018 to 2025. section 16.5

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains hardware, batching, budget, data, and deployment tradeoffs, while stopping short of formal scaling laws or derivations.
Examples
Uses small-batch training, GPT-3's compute budget, cheaper deployment hardware, and matrix-based geometric scaling as examples.
What it assumes
Assumes basic familiarity with datasets, computation, and models while explaining scalability from parallel hardware and batching upward.
What you can do after
Evaluate whether to scale data, model size, or computation while accounting for hardware cost, data quality, and generalization limits.
How it connects
Connects scalability with deep learning, GPU parallelization, Moore's law, foundation models, reinforcement learning, and test-time adaptation.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 27 books explain scalability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.