Chapter 12: k-nearest neighbors

Introduction

This chapter introduction previews classification with k-nearest neighbors, feature extraction, regression, and the algorithm’s uses and limitations.

12.1 Classifying oranges vs. grapefruit

This section introduces k-nearest neighbors classification by deciding whether an ambiguous fruit is an orange or a grapefruit.

• Nearest-neighbor fruit classification: Explain how the section classifies an uncertain fruit by looking at nearby examples and choosing the type most common among its three closest neighbors.
  Why it matters: Nearby examples offer a way to classify a fruit when its appearance alone leaves you unsure.

Key terms: k-nearest neighbors (KNN), classification, neighbors

12.2 Building a recommendations system

Shows how a movie recommendation system can use nearby users with similar tastes to suggest movies.

• Recommendations from similar users: Explain how the example recommends movies to Priyanka by finding users with similar tastes and suggesting movies they like.
  Why it matters: It connects the idea of nearby, similar users to a concrete recommendation.

Key terms: recommendations system, similarity, similar taste

12.3 Feature extraction

The section shows how to represent fruit or users as numbers, measure their similarity by distance, and use that similarity to recommend movies.

• Features as Numeric Vectors: Explain how to extract features that represent an item or user as a set of numbers: size and redness for fruit, or movie-category ratings for a user.
  Why it matters: A numerical representation makes it possible to compare things such as fruit and movie tastes in the same way.
• Distance as Similarity: Explain how distance between feature vectors indicates similarity, and why the same distance formula can compare two-feature fruit and five-rating users.
  Why it matters: Distance provides a way to decide which fruits or users are most alike.
• Similarity Based Movie Recommendations: Describe how the section uses similar users’ tastes to recommend movies, and why more ratings can improve those recommendations.
  Why it matters: It connects the numerical similarity comparison to a practical recommendation.

Key terms: features, feature extraction, vectors, distance, Pythagorean formula, dimensions, recommendations system, ratings, similar users

12.4 Regression

The section explains how KNN predicts numerical values by averaging nearby examples, applies this to bread sales, and notes a limitation of distance-based similarity.

• KNN Regression and Classification: Distinguish regression from classification, and explain how averaging the ratings of nearby people predicts a movie rating. State that k is the chosen number of neighbors, not necessarily five.
  Why it matters: It helps you recognize when KNN is predicting a number rather than a category.
• Bakery Sales Prediction: Trace how the bakery example uses a day's features and past sales to predict how many loaves to make.
  Why it matters: The bakery example shows how the same regression idea applies beyond movie ratings.
• Cosine Similarity Motivation: Explain why cosine similarity may compare users better than the distance formula when they share tastes but use different rating scales.
  Why it matters: The choice of similarity measure can change which users KNN considers neighbors.

Key terms: k-nearest neighbors, regression, classification, features, nearest neighbors, loaves sold, distance formula, cosine similarity, vectors

12.5 Picking good features

The section explains why KNN recommendations depend on choosing features that are relevant to the prediction and broad enough to represent users’ tastes.

• Relevant and representative features: Explain why features used to compare users for movie recommendations should relate directly to movie taste and should not be biased toward a narrow kind of movie.
  Why it matters: The neighbors KNN finds are only useful if the features used to compare them capture what matters for the recommendation.

Key terms: KNN, features, bias, recommendations engine

12.6 Introduction to machine learning

The section introduces machine learning through KNN and identifies the recommendations system as an example.

• KNN as Machine Learning: Explain that KNN is an introduction to machine learning, which the section describes as making a computer more intelligent, and identify the recommendations system as an example.
  Why it matters: It connects the recommendations example to the broader idea of machine learning.

Key terms: KNN, machine learning, recommendations system

12.7 OCR

The section uses OCR to show how feature extraction and k-nearest neighbors can recognize characters in images, then names the training step.

• Character recognition with neighbors: Explain how OCR can use k-nearest neighbors to recognize a number in a new image.
  Why it matters: It connects a familiar KNN comparison to the task of reading text from an image.
• OCR features and training: Distinguish feature extraction from training in the section’s OCR example.
  Why it matters: It clarifies the two steps the section names without requiring details of how a model is trained.

Key terms: OCR, optical character recognition, KNN, nearest neighbors, feature extraction, training, model, lines, points, curves

12.8 Building a spam filter

This section introduces the Naive Bayes classifier as a way to estimate whether an email is spam from the words it contains.

• Word probabilities in spam filtering: Explain how the section’s simple Naive Bayes spam filter uses an email’s words to estimate whether the email is spam.
  Why it matters: It shows how a classifier can use evidence from an email to make a spam prediction.

Key terms: Naive Bayes classifier, spam, probability

12.9 Predicting the stock market

The section uses stock-market prediction to show why choosing useful features for machine learning can be difficult.

• Limits of stock prediction: Explain why past stock-market patterns, such as yesterday’s movement or a supposed May trend, cannot guarantee a prediction of whether the market will rise or fall.
  Why it matters: It shows why an appealing historical pattern is not necessarily a reliable prediction feature.

Key terms: features, past numbers, future performance, variables

12.10 A high-level overview of training an ML model

This section outlines how to prepare data, train and evaluate an ML model, and tune it before use.

• Data preparation steps: Describe gathering data, removing bad data, and extracting features before training an ML model.
  Why it matters: The model is trained on features drawn from the data you keep.
• Training and held-back evaluation: Explain how the section divides data between training and validation, then evaluates predictions against known answers.
  Why it matters: Checking predictions against held-back answers shows how well the trained model performs.
• Post-evaluation parameter tuning: Describe how evaluating a model can lead to adjusting a parameter and checking whether results improve.
  Why it matters: Evaluation gives you a reason to try an adjustment rather than treating the first trained model as final.

Key terms: data, clean, bad data, features, train, validate, evaluating, predictions, parameter tuning, K

12.11 Recap

The recap summarizes how KNN uses nearby examples for classification and regression, and why comparable numerical features matter.

• KNN prediction types: Distinguish classification from regression and explain that KNN uses the k-nearest neighbors for both.
  Why it matters: This distinction clarifies the two kinds of prediction the recap attributes to KNN.
• Comparable numerical features: Explain what feature extraction produces and why choosing good features matters for KNN.
  Why it matters: KNN depends on representations that let items be compared.

Key terms: KNN, k-nearest neighbors, classification, regression, feature extraction, features