Naive Bayes Classifier

Definition

A Naive Bayes classifier is a probabilistic machine learning algorithm based on Bayes' theorem that predicts a class by assuming all input features are conditionally independent given the class label.

Overview

Naive Bayes applies Bayes' theorem to compute the probability of each possible class given a set of observed features, then predicts whichever class has the highest posterior probability. The 'naive' part of the name comes from its core simplifying assumption: that all features are conditionally independent of one another given the class label. This assumption is almost always technically false in real data — word occurrences in a document, for instance, are clearly correlated — yet the classifier frequently performs surprisingly well in practice despite this violated assumption, particularly for text classification. Different variants of Naive Bayes suit different types of input data. Multinomial Naive Bayes is commonly used for text data represented as word counts or term frequencies, such as in spam filtering. Bernoulli Naive Bayes handles binary features, such as whether a word is present or absent in a document. Gaussian Naive Bayes assumes continuous features follow a normal distribution within each class, suiting numerical data. Training is extremely fast and simple, since it only requires estimating feature probabilities per class from frequency counts (with smoothing, typically Laplace/additive smoothing, to handle unseen feature-class combinations) rather than any iterative optimization. Naive Bayes classifiers require relatively little training data compared to many other algorithms, are computationally cheap both to train and to run inference with, and scale well to very high-dimensional feature spaces, such as bag-of-words text representations with tens of thousands of vocabulary terms. These properties made Naive Bayes the classic algorithm for early spam filters and remains a common baseline for text classification, sentiment analysis, and document categorization tasks, prized for its simplicity, speed, and interpretability even when more sophisticated models can achieve higher accuracy.

Key Concepts

• Based directly on Bayes' theorem for computing class posterior probabilities
• Assumes conditional independence of features given the class ('naive' assumption)
• Multinomial, Bernoulli, and Gaussian variants suit different feature types
• Extremely fast, closed-form training via frequency counting, no iterative optimization
• Laplace (additive) smoothing handles unseen feature-class combinations
• Performs well even when the independence assumption is technically violated
• Scales efficiently to very high-dimensional feature spaces (e.g. bag-of-words text)
• Requires comparatively little training data to produce reasonable predictions

Use Cases

Frequently Asked Questions

Related in AI Fundamentals