Bag of Words
Bag of Words is a simple text representation technique that converts a document into a vector of word counts or frequencies, disregarding grammar, word order, and sentence structure while retaining only which words appear and how often.
Definition
Bag of Words is a simple text representation technique that converts a document into a vector of word counts or frequencies, disregarding grammar, word order, and sentence structure while retaining only which words appear and how often.
Overview
Bag of Words is one of the earliest and most foundational techniques for converting unstructured text into a numerical form that machine learning algorithms can process. Given a corpus of documents, a vocabulary is first built from all the unique words appearing across the collection. Each document is then represented as a vector with one dimension per vocabulary word, where the value in each dimension is either a simple count of how many times that word appears in the document, or, in a simplified variant, a binary indicator of whether the word is present at all. The name reflects the technique's core assumption: a document is treated like an unordered 'bag' containing its words, with all information about grammar, word order, and sentence structure discarded. Two documents with identical word counts but completely different word orders (and, potentially, very different meanings) would produce the same or very similar bag-of-words vectors, which is one of the method's most significant limitations. Despite this simplicity, bag-of-words vectors are surprisingly effective for many tasks, particularly text classification and spam detection, where the presence or frequency of certain distinctive words is often a strong enough signal on its own. Bag-of-words representations are also the starting point for more refined techniques like TF-IDF, which reweights the raw counts to emphasize distinctive terms, and can be extended to n-gram models that capture short sequences of consecutive words rather than single words, partially recovering some local word-order information. Bag-of-words representations are typically very high-dimensional and sparse, since the vocabulary of even a modest corpus can run into the tens of thousands of unique words, with most documents containing only a small fraction of them. This sparsity, along with the complete loss of word order and semantic relationships between words, is what eventually motivated the shift toward dense, learned embedding representations and, later, contextual embeddings from transformer models, which capture far richer linguistic structure.
Key Concepts
- Represents documents as vectors of word counts or presence indicators
- Discards word order, grammar, and sentence structure entirely
- Requires building a fixed vocabulary from the training corpus
- Produces high-dimensional, sparse vectors for typical text corpora
- Serves as the foundation for TF-IDF and n-gram model extensions
- Simple, fast, and easy to implement compared to embedding-based methods
- Effective baseline for tasks like spam detection and topic classification
- Cannot capture semantic similarity between words with different spellings