Principal Component Analysis

Definition

Principal Component Analysis (PCA) is a linear dimensionality reduction technique that transforms correlated variables into a smaller set of uncorrelated components, ordered by how much variance in the original data they explain.

Overview

PCA works by finding a new coordinate system for the data where the axes, called principal components, are ordered by the amount of variance they capture. The first principal component points in the direction of greatest variance in the data; the second is orthogonal to the first and captures the next greatest remaining variance, and so on. Mathematically, this is computed via eigenvalue decomposition of the data's covariance matrix, or equivalently through singular value decomposition (SVD) of the (typically mean-centered) data matrix. Because the components are ordered by explained variance, PCA allows practitioners to reduce dimensionality by keeping only the top few components that capture most of the meaningful variation, discarding later components that mostly represent noise. The proportion of total variance explained by the retained components — often visualized with a scree plot — guides how many components to keep, balancing information retention against dimensionality reduction. A key limitation is that PCA is a linear technique: it can only capture linear relationships between variables, so it performs poorly on data with complex, curved (nonlinear) structure, where methods like autoencoders or manifold learning techniques such as t-SNE and UMAP are more effective. PCA is one of the oldest and most widely used techniques in statistics and machine learning, dating back to work by Karl Pearson in 1901 and Harold Hotelling in the 1930s. It's used for exploratory data analysis, noise reduction, data compression, visualization, and as a preprocessing step to speed up and stabilize downstream machine learning models by removing multicollinearity between features. Because principal components are linear combinations of the original features, they can be harder to interpret directly than the original variables, which is a trade-off practitioners weigh against PCA's benefits in dimensionality and noise reduction.

Key Concepts

• Finds orthogonal principal components ordered by explained variance
• Computed via eigenvalue decomposition of the covariance matrix or SVD
• First component captures the direction of greatest variance in the data
• Enables dimensionality reduction by keeping only top components
• A purely linear technique, limited on data with nonlinear structure
• Removes multicollinearity between correlated original features
• Widely used for visualization, noise reduction, and preprocessing
• Dates back to foundational statistical work by Pearson and Hotelling

Use Cases

Frequently Asked Questions

Related in AI Fundamentals