Data transformation

As explained in Fast Python: High performance techniques for large datasets

A data transformation is an operation or function that maps input data or its representation to modified data or a different representation.

A data transformation maps input data or its representation to modified data or another representation. For performance, choose an approach by considering the data and algorithm demands alongside the computing architecture. section 1.4

Compare approaches against available resources, goals, and trade-offs:

• Consider techniques spanning Python’s built-in features, concurrency, NumPy, Cython, pandas, and Apache Arrow. section 1.4
• Account for the system running the code, from a single computer to GPU-enabled, cluster, or cloud environments. section 1.4

Figure 1.4 Underlying hardware architectures must be taken into account when choosing high-performance coding solutions.

Use the hardware overview to consider architecture when choosing a high-performance solution. section 1.4

How Fast Python: High performance techniques for large datasets covers it

How deep it goes

Frames performance decisions around data, algorithms, and architecture without detailing a specific transformation procedure.

Examples

Names Python built-ins, concurrency, NumPy, Cython, pandas, and Apache Arrow as performance-focused areas.

What you can do after

Evaluate and select an efficient solution suited to the system’s resources, goals, and problem.

How it connects

Connects data processing to CPU design, GPUs, storage, network protocols, and cloud architectures.

liveBookmanning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  The Joy of JavaScriptA map is a function or method that applies a transformation to each element or value in a structure and returns a corresponding structure.
• 

  liveBookmanning.com
  Statistics Every Programmer Needsmap is a pandas Series method that applies a mapping or function to each element and returns the transformed values.
• 

  liveBookmanning.com
  Learn Rust in a Month of Lunchesmap is an iterator adaptor that applies a transformation to each item and returns a new iterator producing the transformed items.
• 

  liveBookmanning.com
  Snowflake Data EngineeringData transformation is the conversion of data from a raw source representation into information that satisfies defined requirements.
• 

  liveBookmanning.com
  Platform Engineering on KubernetesA transformation function is a function that maps an incoming event into a different event representation, optionally enriching its data.
• 

  liveBookmanning.com
  Kotlin in Action, 2Emap is a Kotlin standard-library operation that transforms each element and collects the resulting values into a new collection.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA feature transformation maps input feature values to a different representation, often to simplify relationships for a model.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA transformation that maps data or representations within or between geometric spaces.
• 

  liveBookmanning.com
  Effective Data AnalysisA data transformation is a mathematical or computational modification of data values intended to change their representation, scale, distribution, or modeled relationship.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA mapping between geometric spaces, used here as the view of a machine learning model as mapping input points to output points.
• 

  liveBookmanning.com
  F# in Action: A Revised Edition of Get Programming with F#A data transformation is a computation that converts input data into a different value or representation.
• 

  liveBookmanning.com
  Kafka for ArchitectsA map operation transforms each input event into a corresponding event in another format or representation.
• 

  liveBookmanning.com
  Python How-To
• 

  liveBookmanning.com
  Idiomatic Rustmap is an iterator or result-like-type method that applies a function to each available value and returns the transformed result.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentData transformation is the conversion of data into a different structure, representation, or format.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA data transformation is an operation that changes the form or encoding of data while preserving information relevant to a task.
• 

  liveBookmanning.com
  Grokking AI ApplicationsData transformation is the modification, restructuring, cleaning, or reformatting of data to meet processing or storage requirements.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresA map operation transforms each input record or partition into intermediate information relevant to a later computation.
• 

  liveBookmanning.com
  Software Testing with Generative AIThe process of converting data from one structure, representation, or language into another while preserving its intended information.

Compare how 20 books explain data transformation →

Related concepts

• functionuses · A function is a callable program unit that performs an operation, may accept inputs, and can produce a result or side effect.
• iteratorproduces · An iterator is an object or abstraction that provides successive values from a sequence or data source through a traversal protocol.
• arrayuses · An array is an indexed, ordered collection of elements arranged in one or more dimensions, with elements accessed by position.
• pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• comprehensionalternative to · A comprehension is a language construct that creates a collection by iterating over one or more sources, optionally filtering elements and transforming them.
• iterableuses · An iterable is an object or value that provides or supports a protocol for producing its elements sequentially.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.