Dimensionality Reduction and PCA
Dimensionality reduction techniques compress a dataset with many features into a smaller number of derived features while retaining as much of the meaningful structure in the data as possible. High-dimensional data brings real problems: the 'curse of dimensionality' means distances between points become less meaningful as dimensions grow, models become more prone to overfitting, computation slows down, and visualizing data beyond 3 dimensions is impossible for humans. Principal Component Analysis (PCA) is the most widely used linear dimensionality reduction technique, and it addresses these problems by re-expressing the data along a new set of axes chosen specifically to capture the most variance with the fewest dimensions.
Cricket analogy: Tracking fifty stats per delivery (line, length, seam angle, wind speed...) makes it hard to compare deliveries meaningfully, the curse of dimensionality; PCA compresses these into a couple of composite 'aggression' and 'control' axes that capture most of what matters.
How PCA Works
PCA finds a new set of orthogonal (mutually uncorrelated) axes, called principal components, ordered by how much variance in the original data they capture. The first principal component is the direction along which the data varies the most; the second component is the direction of next-most variance, subject to being perpendicular to the first, and so on. Mathematically, PCA is computed via eigendecomposition of the feature covariance matrix (or equivalently, singular value decomposition of the centered data matrix): the eigenvectors become the principal component directions, and their corresponding eigenvalues indicate how much variance each component explains. Because features are re-expressed as linear combinations of the originals, the resulting components are themselves numeric composites rather than any single original, interpretable measurement.
Cricket analogy: PCA's first principal component is like the single axis that best separates aggressive strikers from defensive accumulators, the direction of greatest variance among batters; the second component, perpendicular to the first, might capture a spin-vs-pace preference, each found via eigendecomposition of the stats covariance.
Choosing the Number of Components and When to Use PCA
A common way to pick how many components to keep is the 'explained variance ratio' — plotting cumulative variance explained against number of components and choosing enough components to capture, say, 90-95% of total variance (a 'scree plot' or 'elbow plot' visualizes this trade-off). PCA is most useful when features are highly correlated with each other (since correlated features carry redundant information that PCA can compress efficiently), for speeding up downstream models on very wide datasets, for visualizing high-dimensional data in 2D or 3D, and for reducing noise. It is less useful, and can even hurt performance, when features are already largely uncorrelated and individually interpretable and that interpretability matters, or when the relationships in the data are strongly non-linear (PCA only captures linear structure).
Cricket analogy: Plotting cumulative variance explained against number of components, like a scree plot, might show two composite axes already capture 92% of what separates batters, making PCA worthwhile; but if you're studying one specific, already-clear stat like strike rate, compressing it away loses interpretability for no gain.
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits
X, y = load_digits(return_X_y=True) # 64 pixel-intensity features per image
# Always scale before PCA -- PCA is sensitive to feature variance/scale
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=0.95) # keep enough components for 95% of variance
X_reduced = pca.fit_transform(X_scaled)
print('original dimensions:', X.shape[1])
print('reduced dimensions:', X_reduced.shape[1])
print('variance explained per component (first 5):', pca.explained_variance_ratio_[:5].round(3))
print('cumulative variance explained:', pca.explained_variance_ratio_.sum().round(3))A helpful mental image for PCA: imagine a cloud of data points shaped like a cigar, floating at an angle in 3D space. Even though the data technically occupies 3 dimensions, almost all its variation happens along the cigar's long axis. PCA finds that axis (the first principal component) and lets you describe most of each point's position with just that one number instead of three.
PCA must be fit on the training set only (like scaling and feature selection) and then applied via .transform() to validation/test data, to avoid leakage. Also, always scale features before PCA — because PCA maximizes variance captured, an unscaled feature with a naturally larger numeric range will dominate the principal components regardless of its actual importance.
- Dimensionality reduction compresses many features into fewer derived ones, mitigating the curse of dimensionality.
- PCA finds orthogonal principal components ordered by how much variance of the original data they capture.
- Principal components are linear combinations of original features and lose direct interpretability.
- The number of components to keep is often chosen via cumulative explained variance (e.g., 90-95%).
- PCA works best on correlated, linearly-related numeric features and should always follow feature scaling.
- Fit PCA only on training data and apply the same fitted transform to validation/test data to avoid leakage.
Practice what you learned
1. What does the first principal component represent in PCA?
2. Why should you scale features before applying PCA?
3. What is a key limitation of PCA compared to feature selection?
4. How is the number of principal components to retain commonly chosen?
5. PCA is least helpful in which scenario?
Was this page helpful?
You May Also Like
Feature Selection Techniques
Discover filter, wrapper, and embedded methods for choosing the most useful subset of features, reducing overfitting and improving model interpretability.
Feature Scaling and Normalization
Learn why rescaling numeric features onto comparable ranges matters, and how standardization, min-max scaling, and robust scaling each affect model behavior.
k-Means Clustering
An unsupervised algorithm that partitions data into k groups by iteratively assigning points to the nearest centroid and recomputing centroids until convergence.
What Is a Neural Network?
An introduction to neural networks as layered compositions of weighted sums and nonlinear activations, and how they learn through forward passes and training.