NLTK Cheat Sheet
NLTK natural language toolkit reference covering tokenization, stopword removal, stemming, lemmatization, part-of-speech tagging, and named entity chunking.
2 PagesBeginnerApr 8, 2026
Tokenization
Split text into sentences and words.
python
import nltknltk.download("punkt")nltk.download("stopwords")nltk.download("averaged_perceptron_tagger")from nltk.tokenize import word_tokenize, sent_tokenizetext = "NLTK is a leading platform for building Python NLP programs."sentences = sent_tokenize(text)words = word_tokenize(text)
Stopwords, Stemming & Lemmatization
Normalize tokens for downstream tasks.
python
from nltk.corpus import stopwordsfrom nltk.stem import PorterStemmer, WordNetLemmatizerstop_words = set(stopwords.words("english"))filtered = [w for w in words if w.lower() not in stop_words]stemmer = PorterStemmer()stemmed = [stemmer.stem(w) for w in filtered] # e.g. "running" -> "run"lemmatizer = WordNetLemmatizer()lemmas = [lemmatizer.lemmatize(w) for w in filtered] # dictionary-form words
POS Tagging & Named Entities
Tag parts of speech and chunk entities.
python
from nltk import pos_tag, ne_chunktagged = pos_tag(word_tokenize("Apple is looking at buying a UK startup."))# [('Apple', 'NNP'), ('is', 'VBZ'), ('looking', 'VBG'), ...]tree = ne_chunk(tagged) # named entity chunks: PERSON, ORGANIZATION, GPEprint(tree)
Key Modules
Main areas of the NLTK library.
- nltk.tokenize- splits text into sentences/words
- nltk.corpus- built-in corpora and lexicons (stopwords, wordnet, movie_reviews)
- nltk.stem- PorterStemmer, SnowballStemmer, WordNetLemmatizer
- nltk.tag- part-of-speech tagging
- nltk.chunk- shallow parsing / named entity chunking
- nltk.classify- Naive Bayes and other text classifiers
- nltk.sentiment- VADER sentiment analyzer (SentimentIntensityAnalyzer)
N-grams & Collocations
Extract multi-word phrases and rank statistically significant word pairs.
python
from nltk import ngramsfrom nltk.collocations import BigramCollocationFinder, BigramAssocMeasurestokens = word_tokenize(text.lower())bigrams = list(ngrams(tokens, 2))trigrams = list(ngrams(tokens, 3))finder = BigramCollocationFinder.from_words(tokens)finder.apply_freq_filter(3) # ignore rare pairstop_collocations = finder.nbest(BigramAssocMeasures.pmi, 10)
WordNet Synsets & Similarity
Look up senses, definitions, and compute semantic similarity between words.
python
from nltk.corpus import wordnet as wnsynsets = wn.synsets("bank")for s in synsets[:3]: print(s.name(), "-", s.definition())dog = wn.synset("dog.n.01")cat = wn.synset("cat.n.01")print(dog.path_similarity(cat)) # 0-1, based on taxonomy distanceprint(dog.wup_similarity(cat)) # Wu-Palmer similarityprint(dog.hypernyms(), dog.hyponyms())
Naive Bayes Text Classification
Train and evaluate NLTK's built-in classifier on hand-crafted features.
python
import nltk, randomfrom nltk.corpus import movie_reviewsdocs = [(list(movie_reviews.words(f)), c) for c in movie_reviews.categories() for f in movie_reviews.fileids(c)]random.shuffle(docs)all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words())word_features = list(all_words)[:2000]def doc_features(doc): words = set(doc) return {f"contains({w})": (w in words) for w in word_features}featuresets = [(doc_features(d), c) for d, c in docs]train_set, test_set = featuresets[100:], featuresets[:100]classifier = nltk.NaiveBayesClassifier.train(train_set)print(nltk.classify.accuracy(classifier, test_set))classifier.show_most_informative_features(10)
Custom Chunking with RegexpParser
Define your own shallow-parsing grammar to pull noun/verb phrases from POS tags.
python
from nltk import RegexpParsergrammar = r""" NP: {<DT>?<JJ>*<NN.*>+} # determiner? adjectives* noun(s) VP: {<VB.*><NP|PP>*} # verb followed by NP/PP"""chunk_parser = RegexpParser(grammar)tree = chunk_parser.parse(tagged) # tagged = pos_tag(word_tokenize(sentence))for subtree in tree.subtrees(filter=lambda t: t.label() == "NP"): print(subtree)
Text Analysis Tools
Utilities for exploring corpora beyond basic tokenizing/tagging.
- nltk.FreqDist- frequency distribution over tokens; .most_common(n), .plot()
- nltk.Text.concordance()- show every occurrence of a word with surrounding context
- nltk.ConditionalFreqDist- frequency counts conditioned on a category, e.g. word length by genre
- nltk.edit_distance- Levenshtein distance between two strings, used for fuzzy matching
- nltk.collocations.BigramCollocationFinder- rank statistically significant word pairs by PMI/chi-square
- nltk.sentiment.vader.SentimentIntensityAnalyzer- rule-based polarity scoring tuned for social media text
- nltk.corpus.wordnet- lexical database of synsets, hypernyms/hyponyms, and similarity metrics
Pro Tip
Call nltk.download() for each resource you use (punkt, stopwords, wordnet, etc.) once per environment before running tokenizers or taggers — NLTK ships as a thin library with its data downloaded separately, so a fresh install raises LookupError until resources are fetched.
Was this cheat sheet helpful?
Explore Topics
#NLTK#NLTKCheatSheet#DataScience#Beginner#Tokenization#StopwordsStemmingLemmatization#POS#Tagging#MachineLearning#CheatSheet#SkillVeris
