Data transformation
A data transformation is an operation or function that maps input data or its representation to modified data or a different representation.
The Joy of JavaScript
A map is a function or method that applies a transformation to each element or value in a structure and returns a corresponding structure.
- How it is used
- Transforms array elements, contained values, and observable events as stages in composable data pipelines.
- How deep it goes
- Works through array pipelines, the map/compose correspondence, shared object contracts, and an observable operator implementation; leaves full ADT theory aside.
- Examples
- Uses block validation, wallet-balance calculation, mapped Success values, and a number-sequence observable.
- What you can do after
- Build it from scratch as an observable operator, or apply it to compose array and object transformations.
Read how The Joy of JavaScript explains data transformation →
Statistics Every Programmer Needs
map is a pandas Series method that applies a mapping or function to each element and returns the transformed values.
- How it is used
- Prepares categorical outcomes and predictors as numeric values for logistic regression, decision trees, and random forests.
- How deep it goes
- Demonstrates dictionary-based Series mapping and output checks, then distinguishes encoding from conditional and distribution-shaping transformations.
- Examples
- Uses raisin-variety labels mapped to 0 and 1, plus football venue and play-type labels mapped to numeric codes.
- What you can do after
- Apply it via pandas `map()` to encode categories, then inspect the resulting values and non-null counts before modeling.
Read how Statistics Every Programmer Needs explains data transformation →
Learn Rust in a Month of Lunches
map is an iterator adaptor that applies a transformation to each item and returns a new iterator producing the transformed items.
- How it is used
- Transforms each iterator item into an item for a new iterator, commonly collected into a vector.
- How deep it goes
- Explains closure inputs, ownership choices, lazy execution, and collection; also shows the generic `FnMut` signature.
- Examples
- Doubles numbers into a vector, maps borrowed and owned vector items, and converts an array of integers to strings.
- What you can do after
- Apply it via Rust’s `.map()` and `.collect()` to create transformed vectors, or use array `.map()` to produce a new array.
Read how Learn Rust in a Month of Lunches explains data transformation →
Snowflake Data Engineering
Data transformation is the conversion of data from a raw source representation into information that satisfies defined requirements.
- How it is used
- Transforms ingested Snowflake data into consumer-ready information between ingestion and presentation.
- How deep it goes
- Progresses from SQL summaries to JSON flattening, stored procedures, validation, exception handling, and logging.
- Examples
- Uses bakery CSV orders, staging tables, COPY, MERGE, reporting summaries, scheduled tasks, and JSON files.
- What you can do after
- Build and automate Snowflake transformations using SQL, staging tables, MERGE, transformed tables, and scheduled tasks.
Read how Snowflake Data Engineering explains data transformation →
Platform Engineering on Kubernetes
A transformation function is a function that maps an incoming event into a different event representation, optionally enriching its data.
- How it is used
- Converts collected CloudEvents into CDEvents that metric calculations can consume.
- How deep it goes
- Works through a concrete event mapping and an asynchronous, scheduled function architecture, but does not show transformation code in the supplied evidence.
- Examples
- Maps a Knative API server Deployment event to a service-deployed CDEvent and uses a Kubernetes-based deployment-frequency example.
- What you can do after
- Operate/configure it by scheduling transformation functions and feeding their CDEvents into deployment-frequency calculations.
Read how Platform Engineering on Kubernetes explains data transformation →
Kotlin in Action, 2E
map is a Kotlin standard-library operation that transforms each element and collects the resulting values into a new collection.
- How it is used
- Transforms each collection element into a new form as part of functional collection manipulation.
- How deep it goes
- Defines element-wise transformation, contrasts filtering, and demonstrates lambdas, member references, chaining, indexed mapping, and map values.
- Examples
- Demonstrates squaring numbers, extracting Person names, filtering by age before mapping, indexed mapping, and uppercasing map values.
- What you can do after
- Apply it via the Kotlin standard library to replace manual element-by-element collection transformations.
Read how Kotlin in Action, 2E explains data transformation →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A feature transformation maps input feature values to a different representation, often to simplify relationships for a model.
- How it is used
- Uses feature transformations to prepare classical machine-learning inputs and express nonlinearities or interactions more effectively.
- How deep it goes
- Explains transformation purposes and several strategies, then uses partial dependence to diagnose relationships without deriving transformation mathematics.
- Examples
- Uses linear-form transformations, binning, discretization, precomputed interactions, partial dependence plots, and noisy nonlinear-function model comparisons.
- What you can do after
- Diagnose nonlinearities with partial dependence plots and choose transformations that support model representation and split efficiency.
Deep Learning with Python, Third Edition
A transformation that maps data or representations within or between geometric spaces.
- How it is used
- Interprets deep learning as successive geometric changes that make complex data easier to represent or separate.
- How deep it goes
- Explains layered transformations through a paper analogy and identifies tensor operations, without deriving the geometry mathematically.
- Examples
- Uses crumpled colored paper to illustrate class separation and a Dataset reshape to illustrate specified preprocessing.
- What you can do after
- Evaluate/understand a deep network as composed learned transformations, while distinguishing them from specified dataset preprocessing.
Read how Deep Learning with Python, Third Edition explains data transformation →
Effective Data Analysis
A data transformation is a mathematical or computational modification of data values intended to change their representation, scale, distribution, or modeled relationship.
- How it is used
- Uses transformations to improve statistical assumptions and represent relationships in t-tests and linear regression.
- How deep it goes
- Explains exploratory diagnosis, predictor and outcome transformations, and iterative model evaluation, but does not catalog transformation families.
- Examples
- Demonstrates squared and square-root wind speed, square-root rat sightings, weather outlier removal, and Boston–New York weather testing.
- What you can do after
- Diagnose model problems and apply transformations while reporting what the transformed results can and cannot mean to stakeholders.
Read how Effective Data Analysis explains data transformation →
Math and Architectures of Deep Learning
A mapping between geometric spaces, used here as the view of a machine learning model as mapping input points to output points.
- How it is used
- Uses geometric mappings to place machine-learning data in spaces better suited to classification or simplification.
- How deep it goes
- Defines mappings through vectors and matrices, then connects them to manifolds and neural-network layers without implementation details.
- Examples
- Uses rotation, three-dimensional projection, cylindrical unrolling, and successive neural-network layer mappings.
- What you can do after
- Evaluate machine-learning models as geometric mappings and understand why transformed representations can enable linear classification.
Read how Math and Architectures of Deep Learning explains data transformation →
F# in Action: A Revised Edition of Get Programming with F#
A data transformation is a computation that converts input data into a different value or representation.
- How it is used
- Frames collection processing as pipelines that turn lists of domain data into useful results.
- How deep it goes
- Explains staged immutable collection pipelines and contrasts them with imperative mutation, without detailing every pipeline function initially.
- Examples
- Uses age descriptions and football match results, including filtering away wins, counting teams, and selecting a maximum.
- What you can do after
- Apply it via F# collection pipelines by composing small functions and inspecting intermediate stages in scripts.
Read how F# in Action: A Revised Edition of Get Programming with F# explains data transformation →
Kafka for Architects
A map operation transforms each input event into a corresponding event in another format or representation.
- How it is used
- Frames transformation as an architectural choice for preparing, integrating, and reshaping event data across systems.
- How deep it goes
- Explains stateless mappings, transformation locations, and trade-offs, while deferring complex transformations to streaming frameworks.
- Examples
- Uses type casting, filtering, renaming, format conversion, Customer 360 joins, event mapping, routing, and key changes.
- What you can do after
- Apply it via Kafka Connect or streaming frameworks to reshape, filter, route, join, or aggregate event data.
Read how Kafka for Architects explains data transformation →
Python How-To
- How it is used
- Transforms existing iterable data into converted values or containers suited to later calculations and project use.
- How deep it goes
- Explains map iterators, mapping functions, container construction, and paired zip items; distinguishes map as a class but does not detail its internals.
- Examples
- Converts numeric strings to floats, pairs task IDs with titles, and constructs lists, tuples, dictionaries, and sets from iterables.
- What you can do after
- Apply Python’s map, zip, and container constructors to convert iterable values or build structures from paired data.
Idiomatic Rust
map is an iterator or result-like-type method that applies a function to each available value and returns the transformed result.
- How it is used
- Transforms iterator items or present Option and Result values.
- How deep it goes
- Defines map's closure signature and laziness, then demonstrates collection and Result handling without implementing the iterator.
- Examples
- Demonstrates converting integers to strings, parsing strings, and transforming ranked dog-breed entries.
- What you can do after
- Apply map to transform iterator items or present Option and Result values, then collect or chain the results.
Delivering Machine Learning Projects: From design to deployment
Data transformation is the conversion of data into a different structure, representation, or format.
- How it is used
- Uses transformations to prepare distributed, unreliable data for exploration, model training, evaluation, and reproducible delivery.
- How deep it goes
- Explains aggregation, rebalancing, formatting, filtering, augmentation, and checkpointing, but does not detail transformation algorithms.
- Examples
- Uses consolidated tables or streams, standardized unstructured files, filtered corrupt items, rebalanced data, and augmented training examples.
- What you can do after
- Builds maintainable transformations that produce current, checkpointed, consumable data for model development and evaluation.
Deep Learning with R, Third Edition
A data transformation is an operation that changes the form or encoding of data while preserving information relevant to a task.
- How it is used
- Uses transformed representations to make task-solving rules simpler and less brittle.
- How deep it goes
- Defines several transformation types and frames their selection as hypothesis-space search, without deriving algorithms.
- Examples
- Uses handwritten digits, pixel histograms, closed-loop counts, coordinate changes, projections, translations, and nonlinear selections.
- What you can do after
- Evaluate transformations by how well their representations support simpler rules on development data.
Read how Deep Learning with R, Third Edition explains data transformation →
Grokking AI Applications
Data transformation is the modification, restructuring, cleaning, or reformatting of data to meet processing or storage requirements.
- How it is used
- Applies LLM reasoning to structured business data arriving from files, databases, and enterprise systems.
- How deep it goes
- Explains transformation through KNIME nodes that load, join, sort, aggregate, and reshape data for LLMs.
- Examples
- Uses CRM records, spreadsheets, databases, generated strings, and separated output fields as transformation examples.
- What you can do after
- Apply it via KNIME and Langflow to prepare structured inputs and integrate transformed outputs with downstream systems.
Read how Grokking AI Applications explains data transformation →
Advanced Algorithms and Data Structures
A map operation transforms each input record or partition into intermediate information relevant to a later computation.
- How it is used
- Uses map operations to extract intermediate information before distributed aggregation.
- How deep it goes
- Defines partition-level mapping and its handoff to reduction, while deferring detailed implementation beyond coin recognition and output handling.
- Examples
- Uses coin-recognition output containing unsorted denomination occurrences as the concrete mapped artifact.
- What you can do after
- Apply map operations to extract intermediate records that later reducers can count, sum, or filter.
Read how Advanced Algorithms and Data Structures explains data transformation →
Software Testing with Generative AI
The process of converting data from one structure, representation, or language into another while preserving its intended information.
- How it is used
- Accelerates testing by converting data, code, and notes into formats needed for exploratory testing, debugging, and automation.
- How deep it goes
- Demonstrates reusable prompts and field-completeness checks, but defers final validation because outputs may be inconsistent or invalid.
- Examples
- Uses language translation, JSON-to-SQL conversion, cross-language code conversion, and testing-note summarization as concrete transformations.
- What you can do after
- Apply it via reusable LLM prompts to generate or convert test data while checking structure, relationships, and preserved values.
Read how Software Testing with Generative AI explains data transformation →
Fast Python: High performance techniques for large datasets
- How it is used
- Guides the choice of efficient data-processing solutions by relating algorithm and data demands to available computing resources.
- How deep it goes
- Frames performance decisions around data, algorithms, and architecture without detailing a specific transformation procedure.
- Examples
- Names Python built-ins, concurrency, NumPy, Cython, pandas, and Apache Arrow as performance-focused areas.
- What you can do after
- Evaluate and select an efficient solution suited to the system’s resources, goals, and problem.
Read how Fast Python: High performance techniques for large datasets explains data transformation →
Related concepts
- functionuses · A function is a callable program unit that performs an operation, may accept inputs, and can produce a result or side effect.
- iteratorproduces · An iterator is an object or abstraction that provides successive values from a sequence or data source through a traversal protocol.
- arrayuses · An array is an indexed, ordered collection of elements arranged in one or more dimensions, with elements accessed by position.
- pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- comprehensionalternative to · A comprehension is a language construct that creates a collection by iterating over one or more sources, optionally filtering elements and transforming them.
- iterableuses · An iterable is an object or value that provides or supports a protocol for producing its elements sequentially.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.