Natural Language Processing Basics Cheat Sheet
Foundational NLP techniques including tokenization, stemming, TF-IDF, and word embeddings, with practical code using NLTK, spaCy, and scikit-learn.
2 PagesBeginnerFeb 25, 2026
Text Preprocessing
Tokenize, remove stopwords, stem, and lemmatize.
python
import refrom nltk.corpus import stopwordsfrom nltk.stem import PorterStemmer, WordNetLemmatizerfrom nltk.tokenize import word_tokenizetext = "The cats are running quickly through the gardens!"# Lowercase and remove punctuationtext = re.sub(r"[^\w\s]", "", text.lower())# Tokenizationtokens = word_tokenize(text) # ['the', 'cats', 'are', 'running', ...]# Stopword removalstop_words = set(stopwords.words("english"))tokens = [t for t in tokens if t not in stop_words]# Stemming (crude, rule-based root form)stemmer = PorterStemmer()stems = [stemmer.stem(t) for t in tokens] # 'running' -> 'run'# Lemmatization (dictionary-based, more accurate root form)lemmatizer = WordNetLemmatizer()lemmas = [lemmatizer.lemmatize(t, pos="v") for t in tokens]
TF-IDF & Named Entities
Vectorize text and extract entities with spaCy.
python
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizerimport spacydocs = ["the cat sat on the mat", "the dog sat on the log"]# Bag-of-words countscv = CountVectorizer()X_counts = cv.fit_transform(docs)# TF-IDF: weights terms by importance across the corpustfidf = TfidfVectorizer(max_features=1000, ngram_range=(1, 2))X_tfidf = tfidf.fit_transform(docs)print(tfidf.get_feature_names_out())# spaCy: tokenization, POS tagging, NER, and pretrained word vectorsnlp = spacy.load("en_core_web_sm")doc = nlp("Apple is looking at buying a startup in London.")for ent in doc.ents: print(ent.text, ent.label_) # Apple ORG, London GPE
NLP Concepts
Core vocabulary for text processing.
- Tokenization- splitting text into words, subwords, or sentences
- Stemming- crude rule-based truncation to a word's root (e.g. 'running' -> 'run')
- Lemmatization- dictionary-based reduction to a word's dictionary form, accounts for part of speech
- Stop words- common low-information words (the, is, at) often filtered out
- Bag-of-Words- represents text as unordered word count vectors
- TF-IDF- weights terms by frequency in a document offset by frequency across the corpus, downweighting common words
- Word embeddings- dense vector representations capturing semantic similarity (Word2Vec, GloVe)
- Named Entity Recognition (NER)- identifies and classifies entities like people, organizations, and locations
Common NLP Tasks
Typical problems solved with NLP techniques.
- Text classification- assigning a label to a document, e.g. sentiment or topic
- Named entity recognition- extracting structured entities (people, places, orgs) from text
- Machine translation- converting text from one language to another
- Summarization- producing a shorter version of a document that preserves key information
- Question answering- retrieving or generating an answer to a natural-language question from a context
Subword Tokenization (BPE/WordPiece)
Tokenize with the same subword vocabulary transformer models expect.
python
from tokenizers import Tokenizer, models, trainers, pre_tokenizers# Train a byte-pair-encoding tokenizer from scratchtokenizer = Tokenizer(models.BPE(unk_token="[UNK]"))tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]"])tokenizer.train(files=["corpus.txt"], trainer=trainer)encoded = tokenizer.encode("unhappiness is tokenized subword-wise")print(encoded.tokens) # e.g. ['un', 'happi', 'ness', 'is', ...]# Using a pretrained fast tokenizer (WordPiece for BERT)from transformers import AutoTokenizerbert_tok = AutoTokenizer.from_pretrained("bert-base-uncased")print(bert_tok.tokenize("unhappiness")) # ['un', '##hap', '##pi', '##ness']
Training Word2Vec Embeddings
Learn dense semantic vectors from a corpus with gensim.
python
from gensim.models import Word2Vecsentences = [doc.lower().split() for doc in corpus] # list of tokenized sentencesmodel = Word2Vec( sentences, vector_size=200, window=5, # context window size min_count=5, # ignore rare words sg=1, # 1 = skip-gram, 0 = CBOW negative=10, # negative sampling count epochs=10,)model.wv.most_similar("king", topn=5)model.wv.similarity("king", "queen")# Classic analogy: king - man + woman ~= queenmodel.wv.most_similar(positive=["king", "woman"], negative=["man"], topn=1)
Fine-Tuning a Transformer for Classification
Adapt a pretrained BERT-family model to a downstream text classification task.
python
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainerimport evaluatetokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=2)def tokenize_fn(batch): return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=128)tokenized_ds = dataset.map(tokenize_fn, batched=True)accuracy = evaluate.load("accuracy")def compute_metrics(eval_pred): logits, labels = eval_pred preds = logits.argmax(axis=-1) return accuracy.compute(predictions=preds, references=labels)args = TrainingArguments( output_dir="./out", num_train_epochs=3, per_device_train_batch_size=16, learning_rate=2e-5, eval_strategy="epoch", weight_decay=0.01,)trainer = Trainer(model=model, args=args, train_dataset=tokenized_ds["train"], eval_dataset=tokenized_ds["test"], compute_metrics=compute_metrics)trainer.train()
Topic Modeling with LDA
Discover latent topics across a document collection.
python
from sklearn.decomposition import LatentDirichletAllocationfrom sklearn.feature_extraction.text import CountVectorizervectorizer = CountVectorizer(max_df=0.9, min_df=5, stop_words="english")X = vectorizer.fit_transform(documents)lda = LatentDirichletAllocation(n_components=10, random_state=42, learning_method="online")lda.fit(X)feature_names = vectorizer.get_feature_names_out()for topic_idx, topic in enumerate(lda.components_): top_words = [feature_names[i] for i in topic.argsort()[-10:][::-1]] print(f"Topic {topic_idx}: {', '.join(top_words)}")# Assign the dominant topic to each documentdoc_topics = lda.transform(X).argmax(axis=1)
Advanced NLP Concepts
Terms that come up once you move past classical bag-of-words pipelines.
- Self-attention- mechanism letting each token weigh every other token in the sequence when computing its representation; the core of transformers
- Positional encoding- injects word-order information into transformer inputs since attention itself is order-agnostic
- Perplexity- exponentiated average negative log-likelihood of a language model on held-out text; lower is better
- Byte-Pair Encoding (BPE)- subword tokenization that iteratively merges the most frequent adjacent symbol pairs, balancing vocabulary size and OOV handling
- Contextual embeddings- word representations (e.g. BERT) that change per-sentence based on surrounding context, unlike static Word2Vec/GloVe vectors
- Fine-tuning vs prompting- updating model weights on task-specific data vs. steering a frozen model's behavior purely through input text
- BLEU / ROUGE- n-gram overlap metrics for evaluating machine translation (BLEU) and summarization (ROUGE) against reference text
- Catastrophic forgetting- a fine-tuned model losing previously learned general capabilities when overfit to a narrow new task
Pro Tip
Don't apply aggressive stemming or stopword removal before feeding text into transformer models like BERT — those models rely on subword tokenization and full-sequence context, so preprocessing built for TF-IDF pipelines can actually hurt performance.
Was this cheat sheet helpful?
Explore Topics
#NaturalLanguageProcessingBasics#NaturalLanguageProcessingBasicsCheatSheet#DataScience#Beginner#TextPreprocessing#TFIDFNamedEntities#NLPConcepts#CommonNLPTasks#MachineLearning#CheatSheet#SkillVeris
