Natural Language Processing Basics Cheat Sheet

Foundational NLP techniques including tokenization, stemming, TF-IDF, and word embeddings, with practical code using NLTK, spaCy, and scikit-learn.

2 PagesBeginnerFeb 25, 2026

Text Preprocessing

Tokenize, remove stopwords, stem, and lemmatize.

python

import refrom nltk.corpus import stopwordsfrom nltk.stem import PorterStemmer, WordNetLemmatizerfrom nltk.tokenize import word_tokenizetext = "The cats are running quickly through the gardens!"# Lowercase and remove punctuationtext = re.sub(r"[^\w\s]", "", text.lower())# Tokenizationtokens = word_tokenize(text)  # ['the', 'cats', 'are', 'running', ...]# Stopword removalstop_words = set(stopwords.words("english"))tokens = [t for t in tokens if t not in stop_words]# Stemming (crude, rule-based root form)stemmer = PorterStemmer()stems = [stemmer.stem(t) for t in tokens]        # 'running' -> 'run'# Lemmatization (dictionary-based, more accurate root form)lemmatizer = WordNetLemmatizer()lemmas = [lemmatizer.lemmatize(t, pos="v") for t in tokens]

TF-IDF & Named Entities

Vectorize text and extract entities with spaCy.

python

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizerimport spacydocs = ["the cat sat on the mat", "the dog sat on the log"]# Bag-of-words countscv = CountVectorizer()X_counts = cv.fit_transform(docs)# TF-IDF: weights terms by importance across the corpustfidf = TfidfVectorizer(max_features=1000, ngram_range=(1, 2))X_tfidf = tfidf.fit_transform(docs)print(tfidf.get_feature_names_out())# spaCy: tokenization, POS tagging, NER, and pretrained word vectorsnlp = spacy.load("en_core_web_sm")doc = nlp("Apple is looking at buying a startup in London.")for ent in doc.ents:    print(ent.text, ent.label_)   # Apple ORG, London GPE

NLP Concepts

Core vocabulary for text processing.

Tokenization- splitting text into words, subwords, or sentences
Stemming- crude rule-based truncation to a word's root (e.g. 'running' -> 'run')
Lemmatization- dictionary-based reduction to a word's dictionary form, accounts for part of speech
Stop words- common low-information words (the, is, at) often filtered out
Bag-of-Words- represents text as unordered word count vectors
TF-IDF- weights terms by frequency in a document offset by frequency across the corpus, downweighting common words
Word embeddings- dense vector representations capturing semantic similarity (Word2Vec, GloVe)
Named Entity Recognition (NER)- identifies and classifies entities like people, organizations, and locations

Common NLP Tasks

Typical problems solved with NLP techniques.

Text classification- assigning a label to a document, e.g. sentiment or topic
Named entity recognition- extracting structured entities (people, places, orgs) from text
Machine translation- converting text from one language to another
Summarization- producing a shorter version of a document that preserves key information
Question answering- retrieving or generating an answer to a natural-language question from a context

Pro Tip

Don't apply aggressive stemming or stopword removal before feeding text into transformer models like BERT — those models rely on subword tokenization and full-sequence context, so preprocessing built for TF-IDF pipelines can actually hurt performance.

Was this cheat sheet helpful?

Time Series Analysis Cheat Sheet

Computer Vision Basics Cheat Sheet

Explore Topics

#NaturalLanguageProcessingBasics#NaturalLanguageProcessingBasicsCheatSheet#DataScience#Beginner#TextPreprocessing#TFIDFNamedEntities#NLPConcepts#CommonNLPTasks#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services