liveBook · Manning

Optimization

Also called optimize, optimizing

Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.

How 38 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

An algorithm or object that updates model parameters to reduce a loss function.

How it is used
Updates neural-network parameters to reduce prediction loss during training.
How deep it goes
Derives a gradient-descent update and walks through PyTorch optimizer calls, then extends the loop to interchangeable optimizers and optional schedules.
Examples
Uses quadratic minimization, a batched PyTorch training loop, and contour plots of noisy, exploding, and vanishing gradients.
What you can do after
Apply it via PyTorch optimizers to update model parameters and add an optional learning-rate schedule.

Read how Inside Deep Learning: Math, Algorithms, Models explains optimization →

Think Like a Software Engineering Manager

Continuous improvement is the ongoing refinement of processes, practices, or outcomes through repeated evaluation and adaptation.

How it is used
Uses repeated feedback and adaptation to improve engineering processes, service health, and team experience.
How deep it goes
Provides a practical cycle of small changes, monitoring, feedback, and adjustment rather than a formal optimization method.
Examples
Illustrates improvement through false-positive reports in a code-analysis plug-in and rising on-call workload.
What you can do after
Practice/perform it by choosing a small change, checking its effect, and adapting it with team feedback.

Read how Think Like a Software Engineering Manager explains optimization →

Optimization Algorithms: AI techniques for design, planning, and control problems

An algorithm that assigns values to decision variables to optimize an objective function subject to constraints.

How it is used
Uses optimization to frame design, planning, routing, scheduling, and control as searches for good feasible solutions.
How deep it goes
Develops problem ingredients and search trade-offs, then works through hand iterations and Python solver implementations.
Examples
Illustrates optimization with ticket pricing, cargo-bike loading, and bounded function minimization.
What you can do after
Apply optimization via SciPy’s dual_annealing by defining an objective and bounds and selecting an iteration limit.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains optimization →

Statistics Slam Dunk

Constrained optimization is the process of finding the best value of an objective function subject to specified restrictions.

How it is used
Selects the best feasible combination when budgets, quantities, categories, or aggregate limits restrict available choices.
How deep it goes
Walks through data sorting, binary constraint encoding, matrix construction, lpSolve execution, and verification of a solved roster.
Examples
Uses 24 NBA free agents, position requirements, a $90 million salary cap, and an average-age limit to select five players.
What you can do after
Apply it via lpSolve to maximize or minimize a linear objective while checking hard constraints against the returned solution.

Read how Statistics Slam Dunk explains optimization →

Advanced Algorithms and Data Structures

Optimization is the process of improving a solution or computational procedure according to an objective such as cost, quality, or performance.

How it is used
Frames graph drawing and other large search spaces as searches for solutions with low cost.
How deep it goes
Develops solution encoding, objective choice, search behavior, and implementation trade-offs; does not give one complexity bound for all methods.
Examples
Uses edge-crossing graph drawings, traveling-salesperson routes, and a same-budget comparison of random sampling with simulated annealing.
What you can do after
Practice/perform objective design and method comparison for near-optimal solutions when exhaustive search is impractical.

Read how Advanced Algorithms and Data Structures explains optimization →

Data-Driven Cybersecurity

An iterative process of assessing, planning, implementing, testing, evaluating, and refining changes to tools and processes.

How it is used
Keeps cybersecurity tools, processes, and measures effective as threats and organizational needs change.
How deep it goes
Provides a four-stage improvement cycle, testing guidance, measurable goals, and review practices; does not prescribe a single tool configuration.
Examples
Uses a healthcare phishing response and a before-and-after gap analysis dashboard to show changes and their measured limits.
What you can do after
Practice/perform an improvement cycle by testing targeted changes, comparing results with a baseline, and revising the next action.

Read how Data-Driven Cybersecurity explains optimization →

Deep Learning with R, Third Edition

Optimization is the task of finding input or parameter values that minimize or maximize an objective function.

How it is used
Uses optimization to fit neural-network weights to training data while distinguishing training performance from generalization.
How deep it goes
Explains loss feedback, mini-batch gradients, optimizer configuration, and validation monitoring without deriving optimizer update rules.
Examples
Uses an MNIST digit classifier with Adam and demonstrates RMSprop learning-rate configuration.
What you can do after
Apply optimization via Keras by selecting an optimizer and loss, training a model, and monitoring validation performance.

Read how Deep Learning with R, Third Edition explains optimization →

Effective Conversational AI

Continuous improvement is an iterative cycle of evaluating a system, identifying shortcomings, applying changes, and reassessing results.

How it is used
Guides ongoing chatbot changes toward better user experiences and business outcomes as needs change.
How deep it goes
Lays out a measure–fix–deploy–reassess cycle, then adds prioritization, monitoring, and regression testing without prescribing implementation code.
Examples
Uses misunderstood vaccine questions, a claims-process funnel, and a single-intent fix instead of retraining many intents.
What you can do after
Practice/perform iterative chatbot improvement by selecting a measurable problem, making a focused change, and checking its effect.

Read how Effective Conversational AI explains optimization →

Statistics Every Programmer Needs

An optimization method is an algorithmic procedure for finding a minimum or maximum of an objective function subject to specified constraints.

How it is used
Frames resource allocation and decision-making problems as objective functions optimized within practical constraints.
How deep it goes
Defines objectives and constraints and works through allocation scenarios, but does not provide solver internals or optimization proofs.
Examples
Uses logistics distribution, manufacturing scheduling, project prioritization, and feature portfolios with budget and epic-balance constraints.
What you can do after
Apply it via linear programming to allocate limited resources and evaluate trade-offs among competing objectives.

Read how Statistics Every Programmer Needs explains optimization →

Building LLM Applications with DSPy

The systematic selection or refinement of configurations to improve application performance.

How it is used
Improves language model applications after establishing and evaluating a baseline.
How deep it goes
Explains candidate generation, metric-based validation, and demonstration selection; introduces instruction optimization as a separate approach.
Examples
Uses an airline intent classifier and a four-label flight classification prompt to illustrate optimization.
What you can do after
Apply it via DSPy to improve prompts and compare an optimized application with a baseline.

Read how Building LLM Applications with DSPy explains optimization →

Lead Developer Career Guide

An ongoing practice of identifying and applying incremental changes to improve a product, process, or system.

How it is used
Guides lead developers in improving development workflows and, secondarily, follow-up after difficult conversations.
How deep it goes
Provides review, feedback, communication, and documentation tactics rather than a formal optimization method.
Examples
Contrasts efficient and inefficient organizations and describes process reviews during and after deployments.
What you can do after
Practice/perform recurring process reviews, gather team feedback, communicate proposed changes, and document the reasons for them.

Read how Lead Developer Career Guide explains optimization →

Natural Language Processing in Action, Second Edition

An optimization algorithm is a procedure for selecting parameter values that improve an objective function.

How it is used
Uses gradient-based parameter updates to minimize neural-network loss during training.
How deep it goes
Explains loss surfaces, mini-batch gradients, learning rate, momentum, training loops, and RMSprop, stopping short of deeper optimizer derivations.
Examples
Demonstrates PyTorch SGD training for a weighted name-gender classifier and RMSprop selection for an NLP convolutional neural network.
What you can do after
Apply it via PyTorch by configuring an optimizer, running gradient updates, and monitoring training and test metrics.

Read how Natural Language Processing in Action, Second Edition explains optimization →

Modern C, Third Edition

An optimizer is a compiler component that transforms program code to improve execution or compilation while preserving required behavior.

How it is used
Illustrates how optimization can invalidate measurements of empty loops.
How deep it goes
Compares generated instructions and timings for retained loops without explaining optimizer internals.
Examples
Uses three timed C loops with a volatile counter, a volatile store, and opaque computation.
What you can do after
Evaluate loop measurements by checking whether optimization removed work and whether instrumentation changed its cost.

Read how Modern C, Third Edition explains optimization →

Bayesian Optimization in Action

A method for selecting inputs that maximize or minimize an objective function.

How it is used
Deploys Bayesian optimization when objective optima are costly, dangerous, or otherwise infeasible to use directly.
How deep it goes
Develops feasibility definitions, inequality bounds, Gaussian-process constraint models, acquisition scoring, and BoTorch implementation.
Examples
Uses neural-network architecture costs, drug safety, material production, a one-dimensional toy problem, and four-dimensional airplane design.
What you can do after
Apply it via BoTorch by modeling objectives and constraints, running constrained expected improvement, and comparing feasible optimization progress.

Read how Bayesian Optimization in Action explains optimization →

Timeless Algorithms, The Foundational Ideas

An optimization method or algorithm is a procedure for searching for solutions that improve an objective function.

How it is used
Turns estimation and model training into searches for parameter values that improve a specified objective.
How deep it goes
Connects objectives to iterative updates and constrained formulations, but distinguishes an objective's optimum from future performance.
Examples
Uses handwritten-digit classification with cross-entropy and maximum-margin separation with a hard-margin SVM.
What you can do after
Apply it to maximize likelihood, reduce neural-network loss through gradient updates, or select a margin under stated constraints.

Read how Timeless Algorithms, The Foundational Ideas explains optimization →

Everyday Data Visualization

An iterative development cycle is a repeated process of developing, evaluating, and refining a product across successive phases.

How it is used
Guides improvement through repeated planning, building, and feedback across research, design, and development.
How deep it goes
Details the cycle through project artifacts and gives stopping criteria based on diminishing returns, deadlines, and requirements.
Examples
Uses requirements revisions, wireframe reviews, and an unfinished dashboard whose layout improved after feedback.
What you can do after
Practice/perform repeated feedback rounds, then decide when requirements are met and further changes offer too little benefit.

Read how Everyday Data Visualization explains optimization →

Build AI Drug Discovery Pipelines

An optimizer is an algorithm that updates learned model parameters using loss gradients and an optimization rule.

How it is used
Trains regression and neural-network models by iteratively adjusting parameters to reduce prediction loss.
How deep it goes
Explains batch-level forward passes, loss calculation, backpropagation, updates, and validation logging, but does not derive optimizer algorithms.
Examples
Uses solubility regression, neural-network training batches, MSE, logged epoch losses, and PyTorch’s optimizer methods.
What you can do after
Apply it via PyTorch by configuring an optimizer, running training steps, and monitoring validation loss.

Read how Build AI Drug Discovery Pipelines explains optimization →

Deep Learning with Python, Third Edition

The procedure that updates model parameters based on the loss, typically by a variant of stochastic gradient descent.

How it is used
Guides neural-network training by turning loss feedback into weight updates through a chosen stochastic-gradient-descent variant.
How deep it goes
Explains loss feedback, derivatives, gradients, mini-batches, and named optimizers, while deferring custom training-loop implementation.
Examples
Uses MNIST classification with Keras, Adam, sparse categorical crossentropy, five epochs, and batches of 128 samples.
What you can do after
Operate/configure Keras training with built-in optimizers and learning-rate settings.

Read how Deep Learning with Python, Third Edition explains optimization →

Grokking Machine Learning

An optimization algorithm that updates model parameters to minimize a loss or objective function.

How it is used
Uses an optimizer to train a neural network by selecting a parameter-update algorithm during Keras model compilation.
How deep it goes
Defines the optimizer’s role and names alternatives, but does not explain update rules or compare their mechanics.
Examples
Uses Adam in a Keras classifier and suggests rerunning training with SGD, RMSProp, and AdaGrad.
What you can do after
Apply it via Keras by selecting an optimizer at compilation and comparing training results across alternatives.

Read how Grokking Machine Learning explains optimization →

First-Party Data Activation

The systematic adjustment of decisions or assignments to improve desired outcomes under specified objectives and constraints.

How it is used
Uses experiments and measurement to identify winning audiences, improve campaigns, and balance customer value against marketing fatigue.
How deep it goes
Develops a recurring measure–insight–refine–experiment loop without specifying technical implementation.
Examples
Uses audience saturation, paid-media campaign refinement, and a retailer test reporting 32% higher revenue for personalized recommendations.
What you can do after
Apply measurement findings to refine audience segments, personalization, and channel choices in successive campaigns.

Read how First-Party Data Activation explains optimization →

JavaScript in Depth

A change to an implementation intended to improve execution efficiency while preserving required behavior.

How it is used
Explains how implementation choices affect JavaScript performance without changing specification-level behavior.
How deep it goes
Descends into object representations, optimization tiers, on-stack replacement, type assumptions, and deoptimization triggers.
Examples
Uses predictable object shapes, hot loops, late error branches, integer arithmetic, and coercion-heavy numeric sorting.
What you can do after
Diagnose performance cliffs by relating slow paths and deoptimization to shapes, types, branches, and implicit conversions.

Read how JavaScript in Depth explains optimization →

Grokking Relational Database Design

Optimization is the process of improving database structures or operations to increase efficiency and responsiveness.

How it is used
Improves database efficiency and responsiveness while supporting consistency, maintainability, security, and scalability.
How deep it goes
Explains indexing, denormalization, query evidence, and reference tables, but does not detail query-plan internals.
Examples
Uses product-name searches, product-review lookups, email searches, and country reference data as concrete optimization cases.
What you can do after
Apply it via indexes, denormalization, query metrics, and reference tables to improve performance and preserve data validity.

Read how Grokking Relational Database Design explains optimization →

Grokking AI Algorithms, Second Edition

An optimizer is an algorithm that adjusts model parameters to minimize or otherwise improve an objective function.

How it is used
Frames optimization as a way to improve solutions to complex problems with historical data or many possible choices.
How deep it goes
Defines constraints, local and global bests, and search-space growth, but defers detailed optimizer implementation.
Examples
Uses luggage packing, restaurant selection, and constrained knapsack choices to illustrate search and trade-offs.
What you can do after
Evaluate optimization problems by identifying valid solutions, constraints, local optima, and global objectives.

Read how Grokking AI Algorithms, Second Edition explains optimization →

Applied Reinforcement Learning

Optimization is the selection of controllable decisions that maximize or minimize an objective subject to constraints.

How it is used
Frames controllable internal decisions about future outcomes as questions of what a business should do.
How deep it goes
Defines model components and illustrates formulation with a bakery example, without detailing a general solving procedure.
Examples
Uses truck dispatch scheduling and constrained cake production to illustrate cost minimization and profit maximization.
What you can do after
Evaluate/understand it by mapping business optimization to controllable future decisions and recognizing its model components.

Read how Applied Reinforcement Learning explains optimization →

Effective Platform Engineering

Continuous improvement is an iterative practice of making incremental changes based on observed results and feedback.

How it is used
Uses feedback and operational evidence to evolve platform capabilities, delivery practices, and service reliability.
How deep it goes
Provides an iterative workflow using feedback, process models, metrics, automation prototypes, and reliability reviews.
Examples
Uses idea queues, process-flow diagrams, deployment metrics, automation scripts, CI/CD configurations, and reliability reviews.
What you can do after
Apply it via feedback analysis, measurement, process redesign, automation, and SLO-driven reliability decisions.

Read how Effective Platform Engineering explains optimization →

Machine Learning System Design

Optimization is the improvement of a system's performance or resource efficiency through changes to its models, serving process, code, or hardware.

How it is used
Guides serving and inference work toward measurable bottlenecks and appropriate system-level changes.
How deep it goes
Explains profiling and design choices with timing examples, but stops short of implementation details for quantization, pruning, or distillation.
Examples
Uses 200-millisecond latency decomposition, sequential model pipelines, batching comparisons, preprocessing changes, and a 40% speedup.
What you can do after
Diagnose/debug with it by profiling end-to-end latency, selecting the most optimizable component, and matching effort to scale.

Read how Machine Learning System Design explains optimization →

The Well-Grounded Python Developer: How the pros use Python and Flask

Optimization is the deliberate modification of software to improve a measured performance characteristic.

How it is used
Guides performance work when an application has a specific speed requirement and measurable definition of success.
How deep it goes
Explains target-setting, baselining, profiling, focused changes, iteration, and stopping criteria without detailing particular algorithms or profiling tools.
Examples
Uses application features, baseline measurements, execution profiles, the 90/10 rule, and faster hardware or databases as performance considerations.
What you can do after
Evaluate/understand performance work by setting targets, measuring baselines, locating hot code, testing changes, and balancing speed against complexity.

Read how The Well-Grounded Python Developer: How the pros use Python and Flask explains optimization →

Generative AI in Action

Optimization is the improvement of software or model behavior toward a desired objective such as efficiency, quality, or resource use.

How it is used
Improves prompts and GenAI application operations for quality, efficiency, scalability, user experience, and cost control.
How deep it goes
Provides actionable techniques and monitoring concerns but stops short of mathematical derivation or model-internal optimization.
Examples
Uses prompt refinement, token management, caching, monitoring, load testing, load balancing, auto-scaling, quotas, and rate limiting.
What you can do after
Apply it via monitoring, performance testing, prompt refinement, caching, scaling, and usage controls to improve a GenAI application's operation.

Read how Generative AI in Action explains optimization →

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

Optimization is the process of selecting or constructing a solution that minimizes or maximizes an objective under given conditions.

How it is used
Uses optimization to find routes, connect networks, maximize equations, and solve constraint-based puzzles.
How deep it goes
Explains objectives, representations, fitness conversion, and genetic operations through implementations, without formal proofs or extensive complexity analysis.
Examples
Uses subway and Hyperloop networks, a quadratic equation, and the SEND+MORE=MONEY cryptarithm.
What you can do after
Apply it via GeneticAlgorithm by defining chromosomes, fitness, random construction, crossover, and mutation.

Read how Classic Computer Science Problems in Swift: Essential techniques for practicing programmers explains optimization →

Fabulous Adventures in Data Structures and Algorithms

An optimization is a change to an implementation intended to reduce resource usage or improve performance while preserving its result.

How it is used
Uses optimization to meet performance goals by exploiting problem-specific structure or incrementally improving hot spots.
How deep it goes
Develops concrete recursive and representation-level techniques, including edge cases and mutable-object safety, but leaves measurements and asymptotic conclusions unresolved.
Examples
Illustrates empty-quad shortcuts in HashLife and canonical equivalent-node replacement while building a DAWG.
What you can do after
Apply it via recursive algorithms and canonical data structures to reduce work while retaining existing graph queries.

Read how Fabulous Adventures in Data Structures and Algorithms explains optimization →

Math and Architectures of Deep Learning

A training component that updates model parameters according to a chosen optimization rule such as SGD.

How it is used
Frames optimization as the training process that minimizes loss by learning neural-network weights.
How deep it goes
Explains the objective conceptually and implements SGD through batching, gradient clearing, backpropagation, and parameter updates.
Examples
Demonstrates SGD with a PyTorch classifier trained on Statsville height-and-weight clusters.
What you can do after
Apply optimization via PyTorch by configuring SGD and running a gradient-based training loop.

Read how Math and Architectures of Deep Learning explains optimization →

Build a DeepSeek Model (From Scratch)

An optimizer is an algorithm that updates model parameters using gradients to minimize a training objective.

How it is used
Coordinates parameter updates inside the model training loop.
How deep it goes
Explains configuration and the forward-backward-update cycle, but does not detail AdamW’s internal update rule.
Examples
Uses PyTorch AdamW, optimizer.step(), optimizer.zero_grad(), and checkpoints containing optimizer state.
What you can do after
Operate/configure it in a PyTorch training loop by scheduling its learning rate, applying gradients, and saving its state.

Read how Build a DeepSeek Model (From Scratch) explains optimization →

Learn Generative AI with PyTorch

An algorithm that updates model parameters to reduce loss during training.

How it is used
Uses parameter updates to minimize loss while training classifiers and translation models.
How deep it goes
Explains learning rate selection, gradient computation, gradient clearing, and optimizer steps, without deriving Adam internally.
Examples
Demonstrates Adam with binary image classification and discusses optimizer selection for English-to-French Transformer translation.
What you can do after
Apply it via PyTorch by configuring Adam and stepping through a loss-based training loop.

Read how Learn Generative AI with PyTorch explains optimization →

Machines that Think

Optimization is the process of adjusting a model's parameters according to specified rules to reduce error or improve an objective.

How it is used
Uses parameter updates to reduce neural-network error and frames model behavior under objectives as statistical optimization.
How deep it goes
Explains gradient descent, learning-rate tradeoffs, and momentum qualitatively, then connects optimization to action under competing objectives.
Examples
Uses a hot-and-cold search analogy, a loss-landscape figure, and examples of actions such as trades and resource allocation.
What you can do after
Understand how update rules trade speed against stability and interpret goal-directed language as optimization under pressure, not proof of motives.

Read how Machines that Think explains optimization →

The Art of AI Product Development

Optimization is the systematic adjustment of model, data, or algorithm choices to improve an objective measure of performance.

How it is used
Guides iterative language-model development and deployment toward acceptance criteria, user expectations, and application stakes.
How deep it goes
Details error analysis, data refinement, customization, and operational automation, but stops short of implementation mechanics.
Examples
Uses consumer and medical applications, correcting examples, RAG databases, few-shot examples, and fine-tuning.
What you can do after
Operate/configure a language-model setup through error analysis, data refinement, evaluation, monitoring, and iterative updates.

Read how The Art of AI Product Development explains optimization →

Data Without Labels

Optimization is the systematic selection of values, actions, or configurations to maximize or minimize a defined objective.

How it is used
Guides deep-learning training by adjusting parameters to minimize loss and improve predictive accuracy.
How deep it goes
Explains loss minimization, gradient direction, update frequency, and batch-size tradeoffs, but does not detail adaptive algorithms.
Examples
Compares batch, stochastic, and mini-batch gradient descent and illustrates descent toward a global minimum.
What you can do after
Evaluate optimization choices by comparing update speed, dataset usage, accuracy tradeoffs, and batch-size settings.

Read how Data Without Labels explains optimization →

Building Specialized AI Systems

An optimizer is an algorithm that adjusts model parameters using computed gradients to minimize a training objective.

How it is used
Applies gradient-based parameter updates during language-model fine-tuning.
How deep it goes
Explains learning rates, mini-batch gradients, optimizer variants, and the update loop without deriving optimizer internals.
Examples
Uses language-model prompt-response batches and a training-loop listing with forward, loss, backward, and optimizer-step operations.
What you can do after
Operate/configure training with SGD, Adam, AdamW, or Adafactor and implement the optimizer step in a fine-tuning loop.

Read how Building Specialized AI Systems explains optimization →

Rearchitecting LLMs

An optimization technique is a method for improving a model's efficiency or task performance through changes to its design, parameters, or configuration.

How it is used
Targets efficiency gains without sacrificing performance, including improvements in specific fields.
How deep it goes
Frames architecture-aware optimization as a research capability but gives no specific method or implementation workflow here.
Examples
Uses DistilGPT2 to study Transformer structure and neuron elimination to modify unwanted model behavior.
What you can do after
Build from scratch architecture-aware optimization techniques and implement current research methods.

Read how Rearchitecting LLMs explains optimization →

Also covered in

What to know first

Related concepts

Questions about optimization

What is optimization?

Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.

Which Manning books explain optimization?

Inside Deep Learning: Math, Algorithms, Models, Think Like a Software Engineering Manager, Optimization Algorithms: AI techniques for design, planning, and control problems, Statistics Slam Dunk, Advanced Algorithms and Data Structures, Data-Driven Cybersecurity, Deep Learning with R, Third Edition, Effective Conversational AI and 30 more.

What should I know before learning optimization?

Manning books that teach optimization usually introduce language model, performance, machine learning and global optimum first.

What kinds of optimization are there?

Kinds of optimization include adam, genetic algorithm and linear programming.

What is optimization also called?

Optimization is also called optimize or optimizing.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.