liveBookmanning.com

What you will learn in Designing Deep Learning Systems: A guide for software engineers

A software engineer's guide

by Kit Pang Szeto,Chi Wang

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• Model ServingA production service that hosts a machine learning model and accepts requests to generate predictions.
• model training serviceA service that provides computation resources and job management for executing model-training code, producing trained models, and tracking training execution.
• hyperparameter tuningHyperparameter optimization in which software algorithms select values and manage trials with minimal human intervention.
• workflow orchestrationThe coordination and scheduling of the steps, dependencies, and execution of computational workflows.
• dataset management serviceA system service that stores, organizes, governs, and provides access to machine learning datasets.
• deep learning systemA layered collection of infrastructure components that automates and supports deep learning model and data workflows.
• schemaA data schema that imposes explicit data types and structural constraints on its fields and values.
• metadataData that describes an artifact, experiment, execution, or relationship between machine learning artifacts.
• reproducibilityThe ability to repeat model training with the same inputs and obtain comparable results.
• hyperparameter search spaceThe set of allowable values or value ranges from which hyperparameters may be selected during optimization.
• deep learning development cycleA product-development workflow that moves a deep learning system from business opportunity through data exploration, prototyping, productionization, and application integration.
• metadata storeA persistence service that stores descriptive information about models, including their identifiers, versions, names, file locations, training code, and serving code.
• datasetA unit of collected data that may be related and is managed as a resource for exploration or model training.
• prototypingPrototyping is an experimental phase for evaluating candidate algorithms and building proof-of-concept models against a business requirement.
• commitAn immutable recorded unit of dataset ingestion or change, containing newly added content and associated metadata such as tags and messages.
• metadata and artifacts storeA component that stores artifacts and descriptive metadata, including relationships among datasets, code, models, inferences, and metrics.
• versioningThe practice of recording and retrieving distinct versions of data, code, configurations, or other artifacts over time.
• directed acyclic graphA directed acyclic graph is a directed graph whose edges encode dependencies and whose directed paths contain no cycles.
• distributed trainingDistributed training is the process of training a machine learning model across multiple computing devices or processes, often with data partitioned among workers.
• data parallelA distributed training strategy that replicates a model across computing devices, partitions data batches among them, and combines their parameter updates.
• pipelineA data pipeline is an automated workflow that repeatedly acquires, transforms, labels, or otherwise moves data through processing stages.
• hyperparameter optimization serviceA software library that implements procedures for searching and evaluating machine-learning hyperparameters.
• TorchServeTorchServe is an open-source serving system for deploying and serving PyTorch models in production.
• productionizationThe process of making a machine learning model operational and maintainable in a production environment.
• learning algorithmA procedure that fits a machine learning model to data by optimizing its parameters or objective.
• pipeline parallelismA distributed training technique that partitions a model into sequential stages executed by different workers, allowing batches to flow through the stages concurrently.
• deep learningA machine learning model that uses multiple layers of learned representations to make predictions or decisions.
• resource isolationThe separation of computational workloads to enforce distinct execution environments and resource boundaries.
• dataset reproducibilityDataset reproducibility is the ability to reconstruct or query the same dataset state used at a particular point in a data workflow.
• podA pod is an isolated Kubernetes execution environment that groups application containers and provides a resource guarantee.
• training datasetTraining data is the set of examples from which a machine learning model learns its parameters or behavior.
• ingestion data schemaA schema that defines and validates the structure and representation of data accepted during dataset ingestion.
• training data schemaA schema that defines the format and structure of data used for machine learning model training.
• versioned snapshotAn immutable, version-identified aggregate of selected dataset commits that tracks the preparation status and location of generated training data.
• model parallelismModel parallelism is a distributed training strategy that partitions a model's computation or parameters across multiple devices.
• runA recorded execution of a model-training pipeline that produces outputs and associated parameters, datasets, metrics, artifacts, and metadata.
• data explorationData exploration is the systematic examination of available data to understand its contents, quality, distributions, and useful patterns.
• dataset managerA deep learning system component that collects, organizes, describes, stores, and provides datasets for exploration, labeling, and model training.
• model server approachAn architectural pattern in which a general-purpose model server hosts models and exposes prediction functionality through a common interface.
• data contractAn explicit agreement defining the data structure and expectations shared between an upstream producer and a downstream consumer.
• model service approachA model-serving pattern in which application-specific code contains the model prediction workflow and directly serves a particular model type.
• mlopsA set of practices that bridges machine learning experimentation and production operations to streamline deployment, monitoring, and maintenance.
• container imageA Docker image is an immutable package containing filesystem layers and metadata used to create containers.
• job queueA queue or queue-like collection that stores training jobs awaiting execution or transition to a later execution state.
• Kubeflow training operatorA Kubeflow controller that launches, monitors, and reconciles Kubernetes resources for machine learning training jobs in a specific framework.
• distributed training groupA set of processes or machines registered to coordinate collective computation in a distributed training run.
• schema validationSchema enforcement is the validation of incoming records against a predefined data schema during writing.
• Delta LakeDelta Lake is an open-source storage layer that adds transactional reliability, metadata management, schema control, and versioning to data stored in object storage.
• interactive data science environmentA remote environment that provides secure interactive tools for exploring data, prototyping models, and troubleshooting deep learning system activity.
• PetastormPetastorm is an open-source library for reading Parquet-based datasets in single-machine or distributed deep learning training and evaluation.
• inference requestA request submitted to a deployed model to generate an inference from supplied input.
• Bayesian optimizationBayesian optimization is a sequential optimization technique that uses a probabilistic surrogate model to select promising evaluations of an expensive objective.
• model-free optimizationAn optimization approach that makes no model-based assumptions about the relationship between evaluated configurations or the training code.
• multifidelity optimizationAn optimization approach that uses cheaper approximations of an objective at different fidelity levels to reduce evaluation cost.
• cloud storageCloud object storage is a scalable storage service that persists data as objects addressed by keys rather than as rows in a conventional filesystem or database.
• NVIDIA Triton Inference ServiceNVIDIA Triton Inference Server is an open-source inference-serving system that supports models from multiple machine-learning frameworks and hardware backends.
• objective functionA function that maps hyperparameter settings and their resulting training outcomes to a scalar value to be minimized or maximized.
• gradient synchronizationThe coordination of gradient values across distributed training workers so their model updates remain consistent.
• random searchA hyperparameter search method that samples configurations randomly from a defined search space until a search budget is exhausted.
• TF_CONFIGAn environment variable that describes cluster and task information required for TensorFlow distributed execution.
• model storeA directory or storage location containing model files that a model-serving runtime can load and register for inference.
• custom resource definitionA Kubernetes API extension that defines a custom resource kind and its schema.
• model training codeExecutable program logic that obtains training inputs, fits a machine learning model, and persists resulting models or training artifacts.
• Ray TuneRay Tune is a library for distributed hyperparameter tuning and experiment execution built on the Ray platform.
• grid searchA hyperparameter search method that evaluates configurations formed from the Cartesian product of specified candidate values.
• Delta tableA Delta Lake table is a versioned tabular data structure stored in object storage with transaction-log metadata, schema enforcement, and support for table operations.
• ParquetApache Parquet is an open column-oriented file format designed for efficient storage and retrieval of structured data.
• time travelTime travel is a data-versioning capability that retrieves a table as it existed at a specified historical point or version.
• web predict APIAn interface for submitting requests that execute a model and return its prediction result.
• machine learningMachine learning is the field of methods that learn patterns or decision functions from data rather than relying solely on explicitly programmed rules.
• data ingestion APIAn interface through which data is submitted to a dataset management system for incorporation into managed datasets.
• concept driftA change over time in the relationship between model inputs and the target behavior being predicted.
• proof-of-concept modelA preliminary machine learning model built to evaluate the feasibility of an approach against a requirement.
• direct model embeddingA model-serving design in which model execution code is integrated with application logic and runs in the same process.
• model developmentThe process of designing, training, and validating a machine learning model.
• DistributedDataParallelPyTorch's DistributedDataParallel library component that runs a model across processes or machines and coordinates distributed updates.
• model artifactA persisted file containing the data required to reconstruct or execute a trained machine learning model.
• manifest fileA file that records metadata or associations describing the contents of a dataset or collection of files.
• use caseA description of how an actor interacts with a system to achieve a goal.
• model managerA software component that loads, initializes, caches, and executes machine learning models.
• multitenancyAn application that serves multiple tenants while keeping their data, configuration, or usage contexts logically separated.
• hyperparameterA configuration value selected before or outside model fitting that controls a learning algorithm or training process.
• model repositoryA model repository is a filesystem-based collection of directories and files that stores deployable model artifacts.
• MLflowMLflow is an open-source platform for managing the machine learning lifecycle, including experiment tracking, artifact management, model registration, and deployment.
• prediction platformA platform that exposes shared infrastructure for serving multiple applications and different types of machine learning models.
• CRD objectA custom resource instance representing a training job and exposing its desired and current state through the Kubernetes API.
• PachydermPachyderm is a Kubernetes-based tool for building version-controlled, automated data pipelines backed by object storage.
• ArgoArgo Workflows is an open-source container-native workflow engine for orchestrating jobs and pipelines, commonly on Kubernetes.
• OptunaOptuna is an open-source Python library for automated hyperparameter optimization with trial pruning, parallel execution, and visualization.
• training stateTraining state is the persisted information required to resume model training from a particular point, including learned parameters and execution progress.
• model registrationThe process of making a model known to a model-serving runtime so it can be loaded and served.
• KubernetesKubernetes is an open-source platform for automating deployment, scaling, and management of containerized applications.
• Kubernetes namespaceA namespace is a Kubernetes virtual-cluster boundary used to partition resources and workloads within a physical cluster.
• provenanceThe record of the source data, transformations, and dependencies associated with an artifact.
• generic datasetA dataset type without an enforced schema or data validation that permits unrestricted data upload and download.
• algorithm reproducibilityThe ability to rerun a machine learning algorithm on a specified dataset and obtain the same or similar results.
• ML MetadataAn open-source library for recording, storing, and querying metadata about machine learning workflows and their artifacts.
• Kubernetes job trackerA service component that launches, monitors, and updates the execution state of training jobs running in Kubernetes.
• graph execution componentA serving-system component that executes a computation graph representing model operations and dependencies.
• routing componentA serving-system component that selects an inference backend for a prediction request according to model metadata and routing configuration.
• HorovodA distributed deep learning framework that provides a common interface for accelerating training across multiple machine-learning frameworks.
• data engineerA specialist who builds and operates systems for collecting, transforming, enriching, labeling, and continuously ingesting data.
• data scientistA data scientist is a practitioner who formulates business problems as machine learning problems and develops, evaluates, and operationalizes data-driven solutions.
• Kubeflow KatibKubeflow Katib is a Kubernetes-native system for automated machine learning experimentation and hyperparameter tuning.
• HyperoptAn open-source Python library for defining and performing hyperparameter optimization experiments.
• schedulerA workflow-system component that detects submitted workflows and dispatches their tasks according to dependency and ordering constraints.
• neural networkA computational model composed of interconnected parameterized units that learns mappings from inputs to outputs.
• model archiveA TorchServe model archive, typically a .mar file, containing the packaged files required to load and serve a model.
• memory storeAn in-memory component that temporarily stores job objects and their execution-state collections.
• tree-structured parzen estimatorTPE is a tree-structured Parzen estimator algorithm for sequential hyperparameter optimization.
• synchronous data parallelismA data-parallel training technique in which workers process separate minibatches and synchronize aggregated gradients before updating replicated models.
• MetaflowMetaflow is an open-source framework for building, executing, and managing data science and machine learning workflows.
• state persistenceThe retention of data across process executions and system interactions so that it remains available for later retrieval.
• containerAn isolated runtime instance created from an image that packages an application's code and dependencies.
• data annotationThe process of assigning annotations or target labels to data examples for supervised machine learning.
• fault toleranceThe ability of a system to continue operating correctly or recover when components or operations fail.
• gRPCAn open-source remote procedure call framework that enables communication between distributed services.
• hyperparameter optimization trial loopA repeated execution cycle that proposes hyperparameters, runs a training trial, records its results, and requests another proposal.
• GPipeA pipeline-parallel training implementation that splits batches into microbatches and accumulates their gradients before updating model parameters.
• gpu utilizationThe proportion of available graphics-processing-unit computation capacity that is active during a workload.
• workflow objectAn Argo Workflows resource representing one workflow instance together with its definition and execution state.
• YAMLA human-readable data serialization format commonly used for structured configuration.
• strongly typed datasetA dataset whose records conform to an explicitly defined schema and data types.
• adaptive request batchingDynamic batching is the runtime aggregation of separately arriving inference requests into batches for joint execution.
• TorchScriptTorchScript is a serializable and optimizable intermediate representation of PyTorch programs that can execute independently of Python.
• Triton backendAn implementation that executes a model of a particular framework or model type within Triton.
• machine learning infrastructureThe software, hardware, and services that support machine learning data processing, training, deployment, and operation.
• sidecar processA container deployed alongside a primary application container to provide supporting functionality such as adaptation, proxying, or initialization.
• predictor backend clientA client component that represents and communicates with a backend predictor service, including transferring models or prediction requests.
• predictor connection managerA component that registers predictor clients and selects a predictor backend based on a model's algorithm type.
• backward compatibilityThe property of a system or data interface that allows newer changes to remain usable by existing consumers and projects.
• bounding boxA rectangular coordinate representation that localizes an object within an image.
• object detectionObject detection is the computer-vision task of locating instances of target object classes in images and assigning labels to them.
• pachctlpachctl is the command-line client for connecting to and managing a Pachyderm workspace.
• evaluation modeA model execution mode that disables training-specific behavior so a neural network can produce inference results.
• asynchronous model updateA distributed optimization method in which workers apply model updates without waiting for other workers to synchronize.
• memory limitationA limitation imposed when the memory required by training data or a model exceeds the capacity of a device.
• synchronous model updateA distributed optimization method in which workers synchronize before applying a shared model update.
• cloud-agnostic architectureAn architecture designed to avoid dependence on a particular cloud provider's infrastructure or services.
• worker groupA worker group is a group of parallel workers that jointly execute a distributed training workload.
• batch sizeThe number of training examples processed together in one model-training step.
• Python annotationAn optional Python syntax feature that attaches metadata to function parameters, return values, variables, or other program elements.
• self.next functionA Metaflow API function that declares the successor step or steps for a workflow step.
• step annotationA Metaflow annotation that marks a function as a workflow step.
• self-defined dataset formatA dataset representation defined independently of any particular model-training framework.
• cache miss ratioThe proportion of cache lookup requests whose requested model is not present in the cache.
• LRUThe least recently used algorithm evicts the item that has gone unused for the longest time when a cache reaches its capacity.
• model typeA category of machine learning models sharing a particular input/output format and data-processing logic.
• tenantA customer organization that uses a shared application while maintaining a separate account and segregated data.
• operatorA workflow component that executes a task according to a defined operation implementation.
• model focusA metadata-system design approach that organizes and correlates metadata primarily around model files.
• pipeline focusA metadata-system design approach that organizes and correlates metadata primarily around pipelines or training runs.
• training experimentA grouping construct that organizes multiple related machine learning training runs.
• client SDKA software development kit that lets client components communicate with a service programmatically.
• data aggregatorA software component that organizes related metadata and routes it to storage or query results.
• metrics objectA metadata entity that stores quantitative measurements produced during a training run.
• models objectA metadata entity that records identity and lifecycle information for persisted model files.
• parameters objectA metadata entity that stores the input parameters and configuration values of a training run.
• data merge operationAn operation that aggregates selected raw dataset commits and converts them into a consolidated training-data file.
• version hash registryAn internal registry that maps training-dataset version hashes to their corresponding versioned snapshots.
• artifact storeA storage system for versioned files and objects produced or consumed by machine learning workflows.
• KServeKServe is an open-source platform that provides standardized interfaces and infrastructure for deploying and serving machine-learning models on Kubernetes.
• regressionA deterioration in a model's measured performance relative to an earlier version or evaluation.
• parameterA numerical value in a model that is adjusted during training to determine its behavior.
• serviceA Kubernetes API resource that provides a stable network endpoint for accessing a set of pods.
• backend predictorA model-specific software component that receives serving inputs and executes prediction logic for a model service.
• ASHA schedulerASHA is a scalable asynchronous successive-halving algorithm for early stopping of underperforming trials and allocating resources to promising trials.
• memory swapMemory swap is a training technique that transfers intermediate neural-network data between CPU memory and GPU memory as computation proceeds.
• model comparisonThe process of comparing machine learning experiments or their resulting models to assess differences in behavior or performance.
• learning rateA learning rate is a hyperparameter that controls the step size used to update model parameters during optimization.
• degradationA decline in a deployed machine learning model's predictive or operational performance over time.
• production onboardingProduction onboarding is the process of moving prototype code into a production system and adapting it for production execution.
• workflow constructionWorkflow construction is the process of decomposing code into computational steps and specifying their ordering and dependencies for orchestrated execution.
• metadata and artifact storeA centralized system that stores machine learning metadata and related artifacts and connects them for discovery and debugging.
• training pipelineAn automated workflow that executes model-training tasks from input data through model production, possibly including deployment.
• Apache AirflowApache Airflow is an open-source platform for authoring, scheduling, and monitoring batch-oriented workflows.
• experimental modelAn experimental model is a candidate machine learning model produced during prototyping for comparison or evaluation.
• master processThe designated coordinating process in a distributed training group, responsible for functions such as worker registration or providing initial model state.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.