liveBook · Manning

PySpark

PySpark is the Python interface to Apache Spark for distributed data processing and analytics.

How 2 Manning books explain it

Software Engineering for Data Scientists

PySpark is the Python interface for Apache Spark's distributed data-processing and analytics engine.

How it is used
Scales exploratory analysis and machine-learning workflows beyond pandas for large datasets.
How deep it goes
Demonstrates setup, data-frame analysis, feature engineering, sampling, and random-forest training, without Spark internals.
Examples
Uses an ads dataset for click-rate grouping, category features, random sampling, train-test splitting, and click prediction.
What you can do after
Apply it to analyze large data frames, engineer features, split datasets, and train a random-forest classifier.

Read how Software Engineering for Data Scientists explains PySpark →

Architecting an Apache Iceberg Lakehouse

PySpark is the Python API for Apache Spark distributed data processing.

How it is used
Uses Python-based Spark code to ingest PostgreSQL data into Apache Iceberg tables stored in an object-backed lakehouse.
How deep it goes
Configures imports, connection variables, packages, extensions, catalogs, and a Spark session, stopping short of full transformation logic.
Examples
Uses a Jupyter notebook, Docker network inspection, MinIO warehouse bucket, and an Iceberg ingestion SparkSession.
What you can do after
Apply PySpark by launching a configured Spark session that can access Nessie and MinIO for Iceberg ingestion.

Read how Architecting an Apache Iceberg Lakehouse explains PySpark →

What to know first

Related concepts

Questions about PySpark

What is PySpark?

PySpark is the Python interface to Apache Spark for distributed data processing and analytics.

Which Manning books explain PySpark?

Software Engineering for Data Scientists and Architecting an Apache Iceberg Lakehouse.

What should I know before learning PySpark?

Manning books that teach PySpark usually introduce Python first.

What is the difference between PySpark and pandas?

PySpark is the Python interface to Apache Spark for distributed data processing and analytics. pandas is an open-source Python library providing labeled data structures and tools for manipulating and analyzing tabular and other structured data.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.