concept list in category pyspark

appears as: list, lists, The list, list
Data Analysis with Python and PySpark

This is an excerpt from Manning's book Data Analysis with Python and PySpark.

Just like in a large-scale factory, you don’t go to each employee and give them a list of tasks. No, here, you’ll provide your list of steps to the manager and let them deal with it. In Spark, the manager/master takes your instructions (carefully written in Python code), translate them in Spark steps and then process them across the worker. The master will also manage which worker (more on them in a bits) has which slice of the data, and make sure that you don’t lose some bits in the process.

2.5  Moving from a sentence to a list of words

When ingesting our selected text into a data frame, PySpark created one record for each line of text, and provided a value column of type String. In order to tokenize each word, we need to split each string into a list of distinct words.

Listing 2.12. Splitting our lines of text into lists of words
from pyspark.sql.functions import col, split

lines = book.select(split(col("value"), " "))

lines

# DataFrame[split(value,  ): array<string>]

lines.printSchema()

# root
#  |-- split(value,  ): array (nullable = true)
#  |    |-- element: string (containsNull = true)

lines.show(5)

# +--------------------+
# |     split(value,  )|
# +--------------------+
# |[The, Project, Gu...|
# |                  []|
# |[This, eBook, is,...|
# |[almost, no, rest...|
# |[re-use, it, unde...|
# +--------------------+
# only showing top 5 rows