Spark
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics across clusters, supporting batch and streaming workloads.
Architecting an Apache Iceberg Lakehouse
Apache Spark is an open-source distributed processing framework for large-scale batch, streaming, and analytical workloads.
- How it is used
- Runs batch, micro-batch, streaming, and transformation pipelines that load and maintain Apache Iceberg tables.
- How deep it goes
- Explains APIs, catalog configuration, writes, merges, streaming, and transformation tradeoffs, but defers detailed Iceberg specification semantics.
- Examples
- Uses PostgreSQL sales data, PySpark notebooks, Docker Compose services, partitioned writes, merges, and Kafka-to-Iceberg streaming.
- What you can do after
- Build ingestion pipelines with PySpark or Spark SQL that read operational or streaming data and write Iceberg tables.
Read how Architecting an Apache Iceberg Lakehouse explains Spark →
AI-Powered Search
Spark is a distributed computing platform for processing large datasets and running scalable data-processing and machine-learning workloads.
- How it is used
- Deploys Spark to generate personalized product recommendations for search from user–item interaction signals.
- How deep it goes
- Demonstrates an end-to-end ALS pipeline, including weighting, filtering, indexing, training, evaluation, cleanup, and search indexing.
- Examples
- Uses product clicks, carts, purchases, RetroTech interactions, ALS recommendations, RMSE evaluation, and a recommendation collection.
- What you can do after
- Build it from scratch by transforming interaction data into ALS recommendations and indexing cleaned user–product results.
AI-Powered Developer: Build software with ChatGPT and Copilot
Apache Spark is an open-source distributed data-processing engine for parallel batch processing, stream processing, and analytics.
- How it is used
- Processes large or streaming datasets for asset tracking, monitoring, analytics, and notification workflows.
- How deep it goes
- Defines Spark architecture and demonstrates a working PySpark streaming adapter with schemas, transformations, UDF registration, sinks, and lifecycle control.
- Examples
- Uses a Kafka asset-location topic, Chicago distance filtering, console output, and a sample asset record located 712.8314662207446 miles away.
- What you can do after
- Build it from scratch as a PySpark adapter that consumes Kafka JSON, applies geographic rules, and streams qualifying records to the console.
Read how AI-Powered Developer: Build software with ChatGPT and Copilot explains Spark →
Data Pipelines with Apache Airflow, Second Edition
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics.
- How it is used
- Offloads large data-processing jobs from Airflow while Airflow starts and monitors their completion.
- How deep it goes
- Explains deployment choices and resource trade-offs but stops short of Spark job implementation details.
- Examples
- Uses Citi Bike and Yellow Cab transformations, with input from S3 and transformed output written back to S3.
- What you can do after
- Operate/configure Spark jobs through SparkSubmitOperator, SSHOperator, SimpleHTTPOperator, or another Airflow trigger.
Read how Data Pipelines with Apache Airflow, Second Edition explains Spark →
Outlier Detection in Python
Apache Spark is an open-source distributed computing engine for processing and analyzing large datasets.
- How it is used
- Supports large-volume data preparation and outlier detection when pandas reaches practical size limits.
- How deep it goes
- Explains dataframe-based univariate testing and preprocessing, but does not detail Spark internals or distributed execution mechanics.
- Examples
- Uses z-score, interquartile-range, median-absolute-deviation, and Isolation Forest workflows on large datasets.
- What you can do after
- Operate/configure PySpark dataframes to test, filter, and prepare large datasets for further outlier analysis.
Also covered in
- Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
- ScyllaDB in Action
- Grokking Concurrency
- Cyber Threat Hunting
- Digital Twins in Action
- Software Engineering for Data ScientistsIn early access (MEAP)
What to know first
- deep learningDeep learning is a machine-learning approach that uses neural networks with multiple computational layers to learn representations from data.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- machine learningMachine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
- ParquetApache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
Related concepts
- Spark Migratorused by · Spark Migrator is a Scylla data-migration tool that uses Apache Spark to move data from Cassandra-compatible databases into ScyllaDB.
- Daskalternative to · Dask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
- PySparkused by · PySpark is the Python interface to Apache Spark for distributed data processing and analytics.
- pipelineuses · A pipeline is an ordered sequence of connected processing stages through which data or work flows, with each stage passing or transforming its output for subsequent stages.
- Apache Iceberguses · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.
Questions about Spark
What is Spark?
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics across clusters, supporting batch and streaming workloads.
Which Manning books explain Spark?
Architecting an Apache Iceberg Lakehouse, AI-Powered Search, AI-Powered Developer: Build software with ChatGPT and Copilot, Data Pipelines with Apache Airflow, Second Edition and Outlier Detection in Python.
What should I know before learning Spark?
Manning books that teach Spark usually introduce deep learning, language model, machine learning and Parquet first.
What is the difference between Spark and Dask?
Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics across clusters, supporting batch and streaming workloads. Dask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
What is Spark a kind of?
Spark is a kind of distributed data-processing platform. A distributed data-processing platform executes data-processing jobs across multiple networked machines.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




