GloVe
By Stanford NLP Group
GloVe (Global Vectors for Word Representation) is a word embedding technique, developed at Stanford, that learns dense vector representations of words by factoring a matrix of global word-to-word co-occurrence statistics computed across an…
Definition
GloVe (Global Vectors for Word Representation) is a word embedding technique, developed at Stanford, that learns dense vector representations of words by factoring a matrix of global word-to-word co-occurrence statistics computed across an entire text corpus.
Overview
GloVe was introduced by Pennington, Socher, and Manning at Stanford in 2014 as an alternative to Word2Vec that combines ideas from two previously separate approaches to learning word representations: local context-window methods like Word2Vec, which predict a word from its immediate neighbors, and global matrix factorization methods, which had traditionally been used in techniques like latent semantic analysis but tended to perform worse than context-window methods on word analogy tasks. GloVe's core insight is that ratios of word co-occurrence probabilities encode meaningful semantic relationships more effectively than raw co-occurrence counts alone. It first constructs a large sparse matrix counting how often each pair of words co-occurs within a certain window across the entire training corpus, then trains a model to learn word vectors whose dot product approximates the logarithm of the words' co-occurrence probability. This lets GloVe leverage global corpus statistics — how often words appear together across the entire dataset — rather than relying only on the local context windows that Word2Vec sees during each individual training step. In practice, GloVe and Word2Vec often produce embeddings of comparable quality, and the choice between them was historically influenced by factors like training corpus size, computational resources, and the availability of pretrained vectors — Stanford released pretrained GloVe vectors trained on large corpora like Wikipedia and Common Crawl, which became widely used as off-the-shelf embeddings for NLP projects that didn't want to train their own. Like Word2Vec, GloVe produces static, context-independent embeddings, a limitation that was later addressed by contextual embedding models such as BERT built on the Transformer architecture. Despite being largely superseded by contextual and LLM-derived embeddings in modern NLP pipelines, GloVe vectors remain a common baseline and teaching example for understanding distributional word representations.
Key Concepts
- Learns word vectors by factoring a global word co-occurrence matrix
- Combines ideas from local context-window and global matrix factorization methods
- Models ratios of co-occurrence probabilities to capture semantic relationships
- Trained on corpus-wide statistics rather than only local context windows
- Stanford released widely used pretrained vectors on Wikipedia and Common Crawl
- Produces static, context-independent word embeddings
- Comparable performance to Word2Vec on many benchmark tasks
- Introduced by Pennington, Socher, and Manning in 2014