Optimization
Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
Inside Deep Learning: Math, Algorithms, Models
An algorithm or object that updates model parameters to reduce a loss function.
- How it is used
- Uses parameter gradients to minimize loss during neural-network training.
- How deep it goes
- Explains construction and loop mechanics, then motivates gradient transformations; defers detailed optimizer variants to later treatment.
- Examples
- Demonstrates quadratic minimization, SGD training of neural networks, AdamW defaults, and FashionMNIST configuration.
- What you can do after
- Operate/configure PyTorch optimizers in batch training loops and substitute update algorithms without rewriting the loop.
Read how Inside Deep Learning: Math, Algorithms, Models explains optimization →
Think Like a Software Engineering Manager
Continuous improvement is the ongoing refinement of processes, practices, or outcomes through repeated evaluation and adaptation.
- How it is used
- Uses repeated feedback and adaptation to improve engineering processes, service health, and team experience.
- How deep it goes
- Provides a practical cycle of small changes, monitoring, feedback, and adjustment rather than a formal optimization method.
- Examples
- Illustrates improvement through false-positive reports in a code-analysis plug-in and rising on-call workload.
- What you can do after
- Practice/perform it by choosing a small change, checking its effect, and adapting it with team feedback.
Read how Think Like a Software Engineering Manager explains optimization →
Optimization Algorithms: AI techniques for design, planning, and control problems
An algorithm that assigns values to decision variables to optimize an objective function subject to constraints.
- How it is used
- Frames design, planning, routing, and control decisions as searches for feasible assignments with good objective values.
- How deep it goes
- Develops variables, objectives, constraints, search trade-offs, hand iterations, and Python solver use; does not establish a single algorithm for every problem.
- Examples
- Uses ticket pricing, cargo-bike loading, timetabling, a bounded quadratic, and continuous-function test problems.
- What you can do after
- Apply it via SciPy’s dual_annealing or a tabu-search solver by defining objectives and bounds, then inspecting and tuning results.
Statistics Slam Dunk
Constrained optimization is the process of finding the best value of an objective function subject to specified restrictions.
- How it is used
- Selects the best feasible combination when budgets, quantities, categories, or aggregate limits restrict available choices.
- How deep it goes
- Walks through data sorting, binary constraint encoding, matrix construction, lpSolve execution, and verification of a solved roster.
- Examples
- Uses 24 NBA free agents, position requirements, a $90 million salary cap, and an average-age limit to select five players.
- What you can do after
- Apply it via lpSolve to maximize or minimize a linear objective while checking hard constraints against the returned solution.
Advanced Algorithms and Data Structures
Optimization is the process of improving a solution or computational procedure according to an objective such as cost, quality, or performance.
- How it is used
- Frames graph drawing and other large search spaces as searches for solutions with low cost.
- How deep it goes
- Develops solution encoding, objective choice, search behavior, and implementation trade-offs; does not give one complexity bound for all methods.
- Examples
- Uses edge-crossing graph drawings, traveling-salesperson routes, and a same-budget comparison of random sampling with simulated annealing.
- What you can do after
- Practice/perform objective design and method comparison for near-optimal solutions when exhaustive search is impractical.
Read how Advanced Algorithms and Data Structures explains optimization →
Data-Driven Cybersecurity
An iterative process of assessing, planning, implementing, testing, evaluating, and refining changes to tools and processes.
- How it is used
- Keeps cybersecurity tools, processes, and measures effective as threats and organizational needs change.
- How deep it goes
- Provides a four-stage improvement cycle, testing guidance, measurable goals, and review practices; does not prescribe a single tool configuration.
- Examples
- Uses a healthcare phishing response and a before-and-after gap analysis dashboard to show changes and their measured limits.
- What you can do after
- Practice/perform an improvement cycle by testing targeted changes, comparing results with a baseline, and revising the next action.
Deep Learning with R, Third Edition
Optimization is the task of finding input or parameter values that minimize or maximize an objective function.
- How it is used
- Uses optimization to fit neural-network weights to training data while distinguishing that aim from performance on unseen data.
- How deep it goes
- Explains loss-driven weight updates and mini-batch training, then configures built-in optimizers; leaves custom training loops for later.
- Examples
- Trains a Dense MNIST digit classifier with Adam, sparse categorical crossentropy, five epochs, and held-out test evaluation.
- What you can do after
- Apply it via Keras compile() and fit() to select an optimizer, train a model, and monitor its results.
Read how Deep Learning with R, Third Edition explains optimization →
Effective Conversational AI
Continuous improvement is an iterative cycle of evaluating a system, identifying shortcomings, applying changes, and reassessing results.
- How it is used
- Guides ongoing chatbot changes toward better user experiences and business outcomes as needs change.
- How deep it goes
- Lays out a measure–fix–deploy–reassess cycle, then adds prioritization, monitoring, and regression testing without prescribing implementation code.
- Examples
- Uses misunderstood vaccine questions, a claims-process funnel, and a single-intent fix instead of retraining many intents.
- What you can do after
- Practice/perform iterative chatbot improvement by selecting a measurable problem, making a focused change, and checking its effect.
Read how Effective Conversational AI explains optimization →
Statistics Every Programmer Needs
An optimization method is an algorithmic procedure for finding a minimum or maximum of an objective function subject to specified constraints.
- How it is used
- Frames resource allocation and decision-making problems as objective functions optimized within practical constraints.
- How deep it goes
- Defines objectives and constraints and works through allocation scenarios, but does not provide solver internals or optimization proofs.
- Examples
- Uses logistics distribution, manufacturing scheduling, project prioritization, and feature portfolios with budget and epic-balance constraints.
- What you can do after
- Apply it via linear programming to allocate limited resources and evaluate trade-offs among competing objectives.
Read how Statistics Every Programmer Needs explains optimization →
Building LLM Applications with DSPy
The systematic selection or refinement of configurations to improve application performance.
- How it is used
- Improves an evaluated LM application by selecting stronger prompts instead of relying on repeated manual revisions.
- How deep it goes
- Describes metric-based candidate testing and demonstration selection, and names optimizers without detailing their algorithms in the supplied evidence.
- Examples
- Uses airline intent classification with ambiguous labels and a scored customer-service prompt comparison.
- What you can do after
- Apply it via DSPy to compare candidate prompts, optimize an application, and evaluate the result against a baseline.
Read how Building LLM Applications with DSPy explains optimization →
Lead Developer Career Guide
An ongoing practice of identifying and applying incremental changes to improve a product, process, or system.
- How it is used
- Guides lead developers in improving development workflows and, secondarily, follow-up after difficult conversations.
- How deep it goes
- Provides review, feedback, communication, and documentation tactics rather than a formal optimization method.
- Examples
- Contrasts efficient and inefficient organizations and describes process reviews during and after deployments.
- What you can do after
- Practice/perform recurring process reviews, gather team feedback, communicate proposed changes, and document the reasons for them.
Read how Lead Developer Career Guide explains optimization →
Natural Language Processing in Action, Second Edition
An optimization algorithm is a procedure for selecting parameter values that improve an objective function.
- How it is used
- Uses gradient-based parameter updates to minimize neural-network loss during training.
- How deep it goes
- Explains loss surfaces, mini-batch gradients, learning rate, momentum, training loops, and RMSprop, stopping short of deeper optimizer derivations.
- Examples
- Demonstrates PyTorch SGD training for a weighted name-gender classifier and RMSprop selection for an NLP convolutional neural network.
- What you can do after
- Apply it via PyTorch by configuring an optimizer, running gradient updates, and monitoring training and test metrics.
Read how Natural Language Processing in Action, Second Edition explains optimization →
Modern C, Third Edition
An optimizer is a compiler component that transforms program code to improve execution or compilation while preserving required behavior.
- How it is used
- Uses optimizer behavior to explain why a timing loop may disappear or measure unintended overhead.
- How deep it goes
- Works through three loop designs, generated assembly, and timing differences without describing optimizer internals.
- Examples
- Compares an empty loop with a volatile counter, a volatile store, and opaque accumulation.
- What you can do after
- Diagnose/debug with it whether optimization invalidates a loop benchmark or changes its overhead.
Bayesian Optimization in Action
A method for selecting inputs that maximize or minimize an objective function.
- How it is used
- Deploys Bayesian optimization when objective optima are costly, dangerous, or otherwise infeasible to use directly.
- How deep it goes
- Develops feasibility definitions, inequality bounds, Gaussian-process constraint models, acquisition scoring, and BoTorch implementation.
- Examples
- Uses neural-network architecture costs, drug safety, material production, a one-dimensional toy problem, and four-dimensional airplane design.
- What you can do after
- Apply it via BoTorch by modeling objectives and constraints, running constrained expected improvement, and comparing feasible optimization progress.
Read how Bayesian Optimization in Action explains optimization →
Everyday Data Visualization
An iterative development cycle is a repeated process of developing, evaluating, and refining a product across successive phases.
- How it is used
- Guides visualization work through repeated planning, building, and stakeholder feedback.
- How deep it goes
- Explains the three-step loop and completion criteria, without detailing implementation mechanics.
- Examples
- Uses a housekeeping dashboard evolving from an uncomfortable draft into a polished version.
- What you can do after
- Apply it to refine a visualization until requirements are met or further changes yield diminishing returns.
Read how Everyday Data Visualization explains optimization →
Deep Learning with Python, Third Edition
The procedure that updates model parameters based on the loss, typically by a variant of stochastic gradient descent.
- How it is used
- Guides neural-network training by turning loss feedback into weight updates through a chosen stochastic-gradient-descent variant.
- How deep it goes
- Explains loss feedback, derivatives, gradients, mini-batches, and named optimizers, while deferring custom training-loop implementation.
- Examples
- Uses MNIST classification with Keras, Adam, sparse categorical crossentropy, five epochs, and batches of 128 samples.
- What you can do after
- Operate/configure Keras training with built-in optimizers and learning-rate settings.
Read how Deep Learning with Python, Third Edition explains optimization →
Grokking Machine Learning
An optimization algorithm that updates model parameters to minimize a loss or objective function.
- How it is used
- Uses an optimizer to train a neural network by selecting a parameter-update algorithm during Keras model compilation.
- How deep it goes
- Defines the optimizer’s role and names alternatives, but does not explain update rules or compare their mechanics.
- Examples
- Uses Adam in a Keras classifier and suggests rerunning training with SGD, RMSProp, and AdaGrad.
- What you can do after
- Apply it via Keras by selecting an optimizer at compilation and comparing training results across alternatives.
First-Party Data Activation
The systematic adjustment of decisions or assignments to improve desired outcomes under specified objectives and constraints.
- How it is used
- Uses experiments and measurement to identify winning audiences, improve campaigns, and balance customer value against marketing fatigue.
- How deep it goes
- Develops a recurring measure–insight–refine–experiment loop without specifying technical implementation.
- Examples
- Uses audience saturation, paid-media campaign refinement, and a retailer test reporting 32% higher revenue for personalized recommendations.
- What you can do after
- Apply measurement findings to refine audience segments, personalization, and channel choices in successive campaigns.
Read how First-Party Data Activation explains optimization →
Grokking Relational Database Design
Optimization is the process of improving database structures or operations to increase efficiency and responsiveness.
- How it is used
- Improves database efficiency and responsiveness while supporting consistency, maintainability, security, and scalability.
- How deep it goes
- Explains indexing, denormalization, query evidence, and reference tables, but does not detail query-plan internals.
- Examples
- Uses product-name searches, product-review lookups, email searches, and country reference data as concrete optimization cases.
- What you can do after
- Apply it via indexes, denormalization, query metrics, and reference tables to improve performance and preserve data validity.
Read how Grokking Relational Database Design explains optimization →
Grokking AI Algorithms, Second Edition
An optimizer is an algorithm that adjusts model parameters to minimize or otherwise improve an objective function.
- How it is used
- Frames optimization as a way to improve solutions to complex problems with historical data or many possible choices.
- How deep it goes
- Defines constraints, local and global bests, and search-space growth, but defers detailed optimizer implementation.
- Examples
- Uses luggage packing, restaurant selection, and constrained knapsack choices to illustrate search and trade-offs.
- What you can do after
- Evaluate optimization problems by identifying valid solutions, constraints, local optima, and global objectives.
Read how Grokking AI Algorithms, Second Edition explains optimization →
Effective Platform Engineering
Continuous improvement is an iterative practice of making incremental changes based on observed results and feedback.
- How it is used
- Uses feedback and operational evidence to evolve platform capabilities, delivery practices, and service reliability.
- How deep it goes
- Provides an iterative workflow using feedback, process models, metrics, automation prototypes, and reliability reviews.
- Examples
- Uses idea queues, process-flow diagrams, deployment metrics, automation scripts, CI/CD configurations, and reliability reviews.
- What you can do after
- Apply it via feedback analysis, measurement, process redesign, automation, and SLO-driven reliability decisions.
Read how Effective Platform Engineering explains optimization →
Machine Learning System Design
Optimization is the improvement of a system's performance or resource efficiency through changes to its models, serving process, code, or hardware.
- How it is used
- Guides serving and inference work toward measurable bottlenecks and appropriate system-level changes.
- How deep it goes
- Explains profiling and design choices with timing examples, but stops short of implementation details for quantization, pruning, or distillation.
- Examples
- Uses 200-millisecond latency decomposition, sequential model pipelines, batching comparisons, preprocessing changes, and a 40% speedup.
- What you can do after
- Diagnose/debug with it by profiling end-to-end latency, selecting the most optimizable component, and matching effort to scale.
Read how Machine Learning System Design explains optimization →
The Well-Grounded Python Developer: How the pros use Python and Flask
Optimization is the deliberate modification of software to improve a measured performance characteristic.
- How it is used
- Guides performance work when an application has a specific speed requirement and measurable definition of success.
- How deep it goes
- Explains target-setting, baselining, profiling, focused changes, iteration, and stopping criteria without detailing particular algorithms or profiling tools.
- Examples
- Uses application features, baseline measurements, execution profiles, the 90/10 rule, and faster hardware or databases as performance considerations.
- What you can do after
- Evaluate/understand performance work by setting targets, measuring baselines, locating hot code, testing changes, and balancing speed against complexity.
Generative AI in Action
Optimization is the improvement of software or model behavior toward a desired objective such as efficiency, quality, or resource use.
- How it is used
- Improves prompts and GenAI application operations for quality, efficiency, scalability, user experience, and cost control.
- How deep it goes
- Provides actionable techniques and monitoring concerns but stops short of mathematical derivation or model-internal optimization.
- Examples
- Uses prompt refinement, token management, caching, monitoring, load testing, load balancing, auto-scaling, quotas, and rate limiting.
- What you can do after
- Apply it via monitoring, performance testing, prompt refinement, caching, scaling, and usage controls to improve a GenAI application's operation.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
Optimization is the process of selecting or constructing a solution that minimizes or maximizes an objective under given conditions.
- How it is used
- Uses optimization to find routes, connect networks, maximize equations, and solve constraint-based puzzles.
- How deep it goes
- Explains objectives, representations, fitness conversion, and genetic operations through implementations, without formal proofs or extensive complexity analysis.
- Examples
- Uses subway and Hyperloop networks, a quadratic equation, and the SEND+MORE=MONEY cryptarithm.
- What you can do after
- Apply it via GeneticAlgorithm by defining chromosomes, fitness, random construction, crossover, and mutation.
Fabulous Adventures in Data Structures and Algorithms
An optimization is a change to an implementation intended to reduce resource usage or improve performance while preserving its result.
- How it is used
- Uses optimization to meet performance goals by exploiting problem-specific structure or incrementally improving hot spots.
- How deep it goes
- Develops concrete recursive and representation-level techniques, including edge cases and mutable-object safety, but leaves measurements and asymptotic conclusions unresolved.
- Examples
- Illustrates empty-quad shortcuts in HashLife and canonical equivalent-node replacement while building a DAWG.
- What you can do after
- Apply it via recursive algorithms and canonical data structures to reduce work while retaining existing graph queries.
Read how Fabulous Adventures in Data Structures and Algorithms explains optimization →
Math and Architectures of Deep Learning
A training component that updates model parameters according to a chosen optimization rule such as SGD.
- How it is used
- Frames optimization as the training process that minimizes loss by learning neural-network weights.
- How deep it goes
- Explains the objective conceptually and implements SGD through batching, gradient clearing, backpropagation, and parameter updates.
- Examples
- Demonstrates SGD with a PyTorch classifier trained on Statsville height-and-weight clusters.
- What you can do after
- Apply optimization via PyTorch by configuring SGD and running a gradient-based training loop.
Read how Math and Architectures of Deep Learning explains optimization →
Learn Generative AI with PyTorch
An algorithm that updates model parameters to reduce loss during training.
- How it is used
- Uses parameter updates to minimize loss while training classifiers and translation models.
- How deep it goes
- Explains learning rate selection, gradient computation, gradient clearing, and optimizer steps, without deriving Adam internally.
- Examples
- Demonstrates Adam with binary image classification and discusses optimizer selection for English-to-French Transformer translation.
- What you can do after
- Apply it via PyTorch by configuring Adam and stepping through a loss-based training loop.
Read how Learn Generative AI with PyTorch explains optimization →
The Art of AI Product Development
Optimization is the systematic adjustment of model, data, or algorithm choices to improve an objective measure of performance.
- How it is used
- Guides iterative language-model development and deployment toward acceptance criteria, user expectations, and application stakes.
- How deep it goes
- Details error analysis, data refinement, customization, and operational automation, but stops short of implementation mechanics.
- Examples
- Uses consumer and medical applications, correcting examples, RAG databases, few-shot examples, and fine-tuning.
- What you can do after
- Operate/configure a language-model setup through error analysis, data refinement, evaluation, monitoring, and iterative updates.
Read how The Art of AI Product Development explains optimization →
Data Without Labels
Optimization is the systematic selection of values, actions, or configurations to maximize or minimize a defined objective.
- How it is used
- Guides deep-learning training by adjusting parameters to minimize loss and improve predictive accuracy.
- How deep it goes
- Explains loss minimization, gradient direction, update frequency, and batch-size tradeoffs, but does not detail adaptive algorithms.
- Examples
- Compares batch, stochastic, and mini-batch gradient descent and illustrates descent toward a global minimum.
- What you can do after
- Evaluate optimization choices by comparing update speed, dataset usage, accuracy tradeoffs, and batch-size settings.
Related concepts
- constraintuses · A constraint is a condition or rule that restricts the permissible states, values, actions, or solutions in a system or problem.
- adamincludes · Adam is an adaptive stochastic-gradient optimization algorithm that updates model parameters using moving estimates of the first and second moments of gradients.
- metricuses · A metric is a quantitative measure used to characterize or assess a property, behavior, performance, or outcome.
- genetic algorithmincludes · A genetic algorithm is an evolutionary optimization method that improves a population of candidate solutions through fitness-based selection and variation, often including recombination and mutation.
- linear programmingincludes · Linear programming is an optimization method that maximizes or minimizes a linear objective subject to linear equality or inequality constraints.
- backpropagationimplements · Backpropagation is a procedure that propagates derivatives backward through a computational graph to calculate how an output or loss changes with respect to intermediate values and parameters.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.