Word2Vec
Word2Vec is a family of neural network-based techniques, introduced by Google researchers in 2013, that learn dense vector representations (word embeddings) of words such that words with similar meanings end up close together in the vector…
Definition
Word2Vec is a family of neural network-based techniques, introduced by Google researchers in 2013, that learn dense vector representations (word embeddings) of words such that words with similar meanings end up close together in the vector space.
Overview
Word2Vec was introduced by Tomas Mikolov and colleagues at Google as a computationally efficient way to learn word embeddings from large amounts of unlabeled text, based on the distributional hypothesis that words appearing in similar contexts tend to have similar meanings. Rather than representing words as sparse, high-dimensional one-hot vectors with no inherent notion of similarity, Word2Vec learns dense, typically 100-to-300-dimensional vectors where semantic relationships are captured geometrically — famously, the vector arithmetic "king" minus "man" plus "woman" lands close to the vector for "queen," demonstrating that the learned space captures meaningful linear relationships between concepts. Word2Vec offers two training architectures. The Continuous Bag of Words (CBOW) model predicts a target word from its surrounding context words, while the Skip-gram model does the reverse, predicting surrounding context words given a target word; Skip-gram tends to perform better on rare words while CBOW trains faster. Both are trained as simple, shallow neural networks using techniques like negative sampling or hierarchical softmax to make training efficient over large vocabularies, without needing any labeled data — the supervision signal comes entirely from word co-occurrence patterns in raw text. Word2Vec was hugely influential in popularizing dense word embeddings as a standard input representation for NLP models, and it was followed by related approaches like GloVe, which uses global co-occurrence statistics rather than local context windows, and fastText, which incorporates subword information to better handle rare and out-of-vocabulary words. However, Word2Vec and its contemporaries produce static embeddings — each word has exactly one vector regardless of context, so a word like "bank" gets the same representation whether it means a riverbank or a financial institution. This limitation was later addressed by contextual embedding models built on the Transformer architecture, such as BERT, which generate different embeddings for a word depending on its surrounding sentence.
Key Concepts
- Learns dense word embeddings from large amounts of unlabeled text
- Based on the distributional hypothesis that context determines meaning
- Offers two architectures: Continuous Bag of Words (CBOW) and Skip-gram
- Captures meaningful linear relationships, like king − man + woman ≈ queen
- Trained efficiently using negative sampling or hierarchical softmax
- Requires no labeled data, only raw text co-occurrence statistics
- Produces static embeddings — one fixed vector per word regardless of context
- Introduced by Tomas Mikolov and colleagues at Google in 2013