Spark
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics across clusters, supporting batch and streaming workloads.
Architecting an Apache Iceberg Lakehouse
Apache Spark is an open-source distributed processing framework for large-scale batch, streaming, and analytical workloads.
- How it is used
- Runs batch, micro-batch, streaming, and transformation pipelines that load and maintain Apache Iceberg tables.
- How deep it goes
- Explains APIs, catalog configuration, writes, merges, streaming, and transformation tradeoffs, but defers detailed Iceberg specification semantics.
- Examples
- Uses PostgreSQL sales data, PySpark notebooks, Docker Compose services, partitioned writes, merges, and Kafka-to-Iceberg streaming.
- What you can do after
- Build ingestion pipelines with PySpark or Spark SQL that read operational or streaming data and write Iceberg tables.
Read how Architecting an Apache Iceberg Lakehouse explains Spark →
AI-Powered Search
Spark is a distributed computing platform for processing large datasets and running scalable data-processing and machine-learning workloads.
- How it is used
- Deploys Spark to generate personalized product recommendations for search from user–item interaction signals.
- How deep it goes
- Demonstrates an end-to-end ALS pipeline, including weighting, filtering, indexing, training, evaluation, cleanup, and search indexing.
- Examples
- Uses product clicks, carts, purchases, RetroTech interactions, ALS recommendations, RMSE evaluation, and a recommendation collection.
- What you can do after
- Build it from scratch by transforming interaction data into ALS recommendations and indexing cleaned user–product results.
AI-Powered Developer: Build software with ChatGPT and Copilot
Apache Spark is an open-source distributed data-processing engine for parallel batch processing, stream processing, and analytics.
- How it is used
- Processes large or streaming datasets for asset tracking, monitoring, analytics, and notification workflows.
- How deep it goes
- Defines Spark architecture and demonstrates a working PySpark streaming adapter with schemas, transformations, UDF registration, sinks, and lifecycle control.
- Examples
- Uses a Kafka asset-location topic, Chicago distance filtering, console output, and a sample asset record located 712.8314662207446 miles away.
- What you can do after
- Build it from scratch as a PySpark adapter that consumes Kafka JSON, applies geographic rules, and streams qualifying records to the console.
Read how AI-Powered Developer: Build software with ChatGPT and Copilot explains Spark →
Data Pipelines with Apache Airflow, Second Edition
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics.
- How it is used
- Offloads large data-processing jobs from Airflow while Airflow starts and monitors their completion.
- How deep it goes
- Explains deployment choices and resource trade-offs but stops short of Spark job implementation details.
- Examples
- Uses Citi Bike and Yellow Cab transformations, with input from S3 and transformed output written back to S3.
- What you can do after
- Operate/configure Spark jobs through SparkSubmitOperator, SSHOperator, SimpleHTTPOperator, or another Airflow trigger.
Read how Data Pipelines with Apache Airflow, Second Edition explains Spark →
Outlier Detection in Python
Apache Spark is an open-source distributed computing engine for processing and analyzing large datasets.
- How it is used
- Supports large-volume data preparation and outlier detection when pandas reaches practical size limits.
- How deep it goes
- Explains dataframe-based univariate testing and preprocessing, but does not detail Spark internals or distributed execution mechanics.
- Examples
- Uses z-score, interquartile-range, median-absolute-deviation, and Isolation Forest workflows on large datasets.
- What you can do after
- Operate/configure PySpark dataframes to test, filter, and prepare large datasets for further outlier analysis.
Related concepts
- Spark Migratorused by · Spark Migrator is a Scylla data-migration tool that uses Apache Spark to move data from Cassandra-compatible databases into ScyllaDB.
- pipelineuses · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
- Daskalternative to · Dask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
- PySparkused by · PySpark is the Python interface to Apache Spark for distributed data processing and analytics.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.