concept batch mode in category pyspark

appears as: batch mode
Data Analysis with Python and PySpark

This is an excerpt from Manning's book Data Analysis with Python and PySpark.

The next two chapters will introduce a basic workflow with PySpark via the creation of a simple ETL (Extract, Transform and Load, which is a more business-speak way of saying Ingest, Transform and Export). We will spend most of our time at the pyspark shell, interactively building our program one step at a time. Just like normal Python development, using the shell or REPL (I’ll use the terms interchangeably) provides rapid feedback and quick progression. Once we are comfortable with the results, we will wrap our program so we can submit it in batch mode.

Our program is looking much more polished now. The last step will be to add the PySpark’s plumbing to prepare it for batch mode.