Naive bayes classifier

As explained in Machine Learning Algorithms in Depth

A probabilistic classifier that assumes features are conditionally independent given the class label.

A naive Bayes classifier maps inputs to discrete class labels by assuming that features are conditionally independent given the class. This assumption factorizes the class-conditional density into one-dimensional feature densities, even when the assumption is not literally true. section 5.5 section 5.8

For Bernoulli naive Bayes, each xᵢⱼ records whether word j is absent or present in document i. Parameters θⱼc describe word probabilities for class c, while π represents the class-label prior. With dictionary size D and C classes, the parameter count is O(DC). section 5.5

Training and inference follow a compact probabilistic derivation:

• Factor the document likelihood across features using conditional independence. section 5.5
• Maximize the log likelihood for the model parameters. section 5.5
• Enforce that class probabilities sum to one with a Lagrangian, then set parameter gradients to zero. section 5.5

Document classification

Represent documents as binary word-presence vectors and classify them with Bernoulli naive Bayes. A from-scratch implementation defines class and dictionary dimensions, initializes π and θ, and fits them from training documents and labels. section 5.5

Figure 5.12 Naive Bayes probabilistic graphical model

Use the probabilistic graphical model to inspect the class label, observed word variables, and conditional-independence structure. section 5.5

Figure 5.13 Naive Bayes algorithm pseudo-code

Use the pseudocode to trace the training procedure before implementing the classifier. section 5.5

How Machine Learning Algorithms in Depth covers it

How deep it goes

Derives Bernoulli training through likelihood maximization and Lagrange constraints, then supplies pseudocode and an implementation.

Examples

Uses binary word-presence vectors, document labels, a graphical model, pseudocode, and a Bernoulli implementation.

What you can do after

Builds a Bernoulli naive Bayes classifier from scratch for document classification.

How it connects

Connects classification with Bernoulli, Gaussian, multinomial, and Dirichlet distributions.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA probabilistic classification model that estimates class probabilities from observed features using the naive Bayes assumption.
• 

  liveBookmanning.com
  Causal AIA naive Bayes classifier is a probabilistic classifier that applies Bayes’ rule while assuming conditional independence among features given the class.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionNaive Bayes is a family of probabilistic classifiers based on Bayes' theorem and a conditional-independence assumption among features.
• 

  liveBookmanning.com
  LLMs in ProductionA naive Bayes classifier is a probabilistic classification model that applies Bayes' theorem while assuming conditional independence among features given the class.

Compare how 5 books explain naive bayes classifier →

Related concepts

• Bayes' theoremuses · Bayes' theorem is a probability rule that expresses a conditional probability in terms of a prior probability, a likelihood, and the evidence.
• bayesian statisticsuses · Bayesian statistics is a framework for statistical inference that combines prior probability distributions with observed-data likelihoods to update probabilities over hypotheses or unknown quantities.
• probability theoryproduces · Probability theory is the branch of mathematics that formalizes uncertainty and analyzes random events and outcomes using probability measures.
• Bernoulli naive Bayesincludes · Bernoulli naive Bayes is a probabilistic classifier that models binary feature occurrences using Bernoulli likelihoods and assumes conditional independence given the class.
• conditional independenceuses · Two variables are conditionally independent given specified information when their conditional joint distribution factors into the product of their conditional distributions.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.