liveBook · Manning

Spark

As explained in AI-Powered Search

Spark is a distributed computing platform for processing large datasets and running scalable data-processing and machine-learning workloads.

Spark supports scalable personalized search by training collaborative-filtering recommendations from user–item interactions. Its built-in ALS implementation learns latent features for users and products. section 9.3.2

A practical Spark recommendation pipeline:

  • Convert clicks, add-to-cart events, and purchases into weighted implicit preferences. section 9.3.2

Weighted implicit preferences

A pipeline can assign clicks a weight of 1 while assigning add-to-cart and purchase events zero weight for a simple ALS setup. Those weights can be increased to represent stronger product interest. section 9.3.2

How AI-Powered Search covers it

How deep it goes
Demonstrates an end-to-end ALS pipeline, including weighting, filtering, indexing, training, evaluation, cleanup, and search indexing.
Examples
Uses product clicks, carts, purchases, RetroTech interactions, ALS recommendations, RMSE evaluation, and a recommendation collection.
What it assumes
Assumes user–item interaction data and teaches signal weighting, identifier indexing, model fitting, and recommendation generation.
What you can do after
Build it from scratch by transforming interaction data into ALS recommendations and indexing cleaned user–product results.
How it connects
Connects Spark with collaborative filtering, matrix factorization, implicit feedback, recommendations, and search-engine ranking.

Also explained in

Compare how 5 books explain Spark →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.