DeepSpeed
DeepSpeed is an open-source deep-learning optimization library for efficient distributed training and inference, particularly of large models.
LLMs in Production
DeepSpeed is an optimization library for distributed deep learning that provides training and inference efficiency features.
- How it is used
- Optimizes distributed deep-learning training and inference across multi-GPU workloads.
- How deep it goes
- Describes major optimizations and installation constraints, but does not explain implementation internals.
- Examples
- Uses multi-GPU workloads, long or multimodal inputs, quantized inference, and pip-based installation as practical examples.
- What you can do after
- Operate/configure DeepSpeed after preparing a compatible CUDA Toolkit and current non-nightly PyTorch installation.
Domain-Specific Small Language Models
An open-source deep learning optimization library that supports distributed training and inference and is used here to speed up and memory-optimize Transformer models.
- How it is used
- Optimizes Transformer inference and distributed training when compute, memory, latency, or hardware capacity constrain deployment.
- How deep it goes
- Explains major mechanisms and demonstrates notebook inference, but does not detail ZeRO internals or distributed training procedures.
- Examples
- Uses GPT-Neo 2.7B, a T4 GPU, float16 loading, CUDA inference, generated text, and latency measurements.
- What you can do after
- Operate/configure DeepSpeed inference for a compatible Transformer model and compare its latency with a baseline.
Read how Domain-Specific Small Language Models explains DeepSpeed →
Related concepts
- librarykind of · A software library is a reusable collection of code or components that other programs can incorporate or invoke to provide functionality.
- PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
- dataset splituses · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
- transformer architectureuses · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- distributed trainingimplements · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.