Machine Learning Chapter 25: Naive Bayes

What this chapter covers

This chapter contains 8 topics. Technical terms are followed by plain-language meanings in parentheses where they first appear. Code is included only when it naturally helps demonstrate the concept; architecture, workflow, governance, and comparison topics use practical scenarios instead.

25.1 Bayes' Theorem

Bayes' Theorem (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Bayes' Theorem to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Bayes' Theorem
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a small real-world example for Bayes' Theorem. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

25.2 Conditional Probability

Conditional Probability (a numerical description of how likely an event is). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Conditional Probability to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Conditional Probability
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a small real-world example for Conditional Probability. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

25.3 Naive Independence Assumption

Naive Independence Assumption (a practical concept used within supervised learning, evaluation, and ensemble methods). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Naive Independence Assumption to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Naive Independence Assumption
const records = [3, 5, 7, 9, 11];
const transform = value => ({ input: value, output: value * 2 + 1 });
const results = records.map(transform);

console.log(results);

Code explanation

The sample starts with a small list of inputs so every result can be checked manually.
`transform()` represents the main operation for this topic in a deliberately simple form.
`map()` applies the same rule consistently to every item and returns a new result array.
Use this pattern to focus on input, transformation, and output before replacing the toy rule with a more advanced method.

Expected result: A transformed result is printed for each input value.

Practice exercise

Create a small real-world example for Naive Independence Assumption. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

25.4 Gaussian Naive Bayes

Gaussian Naive Bayes (a probabilistic classifier based on Bayes theorem and a simplifying independence assumption). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

For spam filtering, words such as 'prize' or 'winner' may occur more often in spam. Naive Bayes combines these probabilities to estimate whether a new email is spam.

Coding example

// Gaussian Naive Bayes
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a second example for Gaussian Naive Bayes. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

25.5 Multinomial Naive Bayes

Multinomial Naive Bayes (a probabilistic classifier based on Bayes theorem and a simplifying independence assumption). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

For spam filtering, words such as 'prize' or 'winner' may occur more often in spam. Naive Bayes combines these probabilities to estimate whether a new email is spam.

Coding example

// Multinomial Naive Bayes
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a second example for Multinomial Naive Bayes. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

25.6 Bernoulli Naive Bayes

Bernoulli Naive Bayes (a probabilistic classifier based on Bayes theorem and a simplifying independence assumption). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

For spam filtering, words such as 'prize' or 'winner' may occur more often in spam. Naive Bayes combines these probabilities to estimate whether a new email is spam.

Coding example

// Bernoulli Naive Bayes
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a second example for Bernoulli Naive Bayes. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

25.7 Text Classification

Text Classification (predicting a category or class). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

An email filter receives a new message and decides whether it belongs to the 'spam' class or the 'not spam' class.

Coding example

// Text Classification
const sigmoid = z => 1 / (1 + Math.exp(-z));
const weights = [0.8, -0.4];
const features = [2, 1];
const bias = -0.2;
const score = weights.reduce((sum, w, i) => sum + w * features[i], bias);
const probability = sigmoid(score);
const predictedClass = probability >= 0.5 ? 1 : 0;

console.log({ probability: probability.toFixed(3), predictedClass });

Code explanation

`weights`, `features`, and `bias` create a simple linear score.
The sigmoid function converts any score into a value between 0 and 1.
A threshold of 0.5 turns the probability into a class label.
Printing both values helps you distinguish a model score from the final classification decision.

Expected result: A probability and a predicted class are printed.

Practice exercise

Create a second example for Text Classification. Change one important condition or input, predict how the result should change, and explain why. Then identify one limitation or common mistake a beginner should watch for.

25.8 Probability Estimates

Probability Estimates (a numerical description of how likely an event is). Within Chapter 25, this topic connects directly to supervised learning, evaluation, and ensemble methods. The important goal is to understand what information goes into the method, what transformation or decision happens, and what output should be checked.

When using this idea, separate training data from evaluation data and compare performance on examples the model did not train on. Pay attention to assumptions, important settings, error patterns, and whether the method is appropriate for classification, regression, ranking, or probability estimation.

Example

Imagine a small machine-learning project. Use Probability Estimates to decide what information is needed, what step happens next, and what result should be checked.

Coding example

// Probability Estimates
const outcomes = [1, 0, 1, 1, 0, 1, 0, 1];
const successes = outcomes.reduce((sum, x) => sum + x, 0);
const probability = successes / outcomes.length;
const smoothed = (successes + 1) / (outcomes.length + 2);

console.log({ probability: probability.toFixed(3), smoothed: smoothed.toFixed(3) });

Code explanation

Each `1` represents an observed success and each `0` represents a non-success.
Dividing the number of successes by the number of observations gives an empirical probability.
The smoothed estimate adds one pseudo-success and one pseudo-failure so very small datasets are less extreme.
Comparing the raw and smoothed results demonstrates how probabilistic estimates can change when prior information is introduced.

Expected result: Two probability estimates are printed for comparison.

Practice exercise

Create a small real-world example for Probability Estimates. Write the input, the goal, the main steps, and the result you would check. Then list one limitation or mistake a beginner should watch for.

Chapter 25 Review Questions and Answers

Q1. What is Bayes' Theorem?

Answer: Bayes' Theorem is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q2. What is Conditional Probability?

Answer: Conditional Probability is a numerical description of how likely an event is. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q3. What is Naive Independence Assumption?

Answer: Naive Independence Assumption is a practical concept used within supervised learning, evaluation, and ensemble methods. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q4. What is Gaussian Naive Bayes?

Answer: Gaussian Naive Bayes is a probabilistic classifier based on Bayes theorem and a simplifying independence assumption. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q5. What is Multinomial Naive Bayes?

Answer: Multinomial Naive Bayes is a probabilistic classifier based on Bayes theorem and a simplifying independence assumption. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q6. What is Bernoulli Naive Bayes?

Answer: Bernoulli Naive Bayes is a probabilistic classifier based on Bayes theorem and a simplifying independence assumption. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q7. What is Text Classification?

Answer: Text Classification is predicting a category or class. In this chapter, focus on the input, the method or decision, and the result that should be checked.

Q8. What is Probability Estimates?

Answer: Probability Estimates is a numerical description of how likely an event is. In this chapter, focus on the input, the method or decision, and the result that should be checked.