What Is Dimensionality Reduction?
Learn what dimensionality reduction is, how PCA, t-SNE, and UMAP work, why the curse of dimensionality matters, and when to reduce your feature set.
Expected Interview Answer
Dimensionality reduction is the process of reducing the number of input features in a dataset while preserving as much meaningful information as possible, making data easier to visualize, faster to process, and less prone to overfitting.
Techniques fall into two broad groups: feature selection, which picks a useful subset of existing features, and feature extraction, which creates new combined features, such as Principal Component Analysis (PCA) projecting data onto directions of maximum variance, or t-SNE and UMAP for non-linear visualization. Reducing dimensions combats the curse of dimensionality, where distances between points become less meaningful and models need exponentially more data as feature count grows, while also cutting storage, training time, and noise from irrelevant or redundant variables.
- Combats the curse of dimensionality in high-feature datasets
- Speeds up training and reduces memory/storage requirements
- Removes redundant or correlated features that add noise
- Enables visualization of high-dimensional data in 2D or 3D
- Can reduce overfitting by simplifying the feature space
AI Mentor Explanation
Dimensionality reduction is like a selector distilling forty raw batting stats down to a handful of key indicators — average, strike rate, and conversion rate — that still capture what makes a player valuable. Instead of drowning in every recorded number from every match, the selector keeps only the combinations of stats that explain most of the variation between players.
How PCA reduces many original features to fewer principal components
Original features
- 50 correlated input variables
Reduction method
- PCA: project onto directions of max variance
- t-SNE/UMAP: non-linear embedding
Reduced features
- 2-10 components capturing ~95% of variance
Step-by-Step Explanation
Step 1
Standardize features
Scale features to comparable ranges, since variance-based methods like PCA are sensitive to scale.
Step 2
Choose a method
Pick feature selection (drop unneeded columns) or feature extraction (PCA, t-SNE, UMAP, autoencoders).
Step 3
Fit the reduction
Compute principal components or embeddings from the training data.
Step 4
Decide how many dimensions to keep
Use explained variance ratio or a scree plot to choose enough components to retain most information.
Step 5
Transform and use
Project the data into the reduced space for modeling, visualization, or storage.
What Interviewer Expects
- Distinguishes feature selection from feature extraction
- Can explain PCA at a high level (variance-maximizing projection)
- Understands the curse of dimensionality and why it matters
- Knows when to use t-SNE/UMAP (visualization) vs. PCA (general-purpose)
- Mentions the need to standardize features before PCA
Common Mistakes
- Applying PCA to unstandardized features with wildly different scales
- Assuming PCA components are directly interpretable like original features
- Using t-SNE output distances as meaningful without understanding its limitations
- Reducing dimensions without checking how much variance is retained
Best Answer (HR Friendly)
“Dimensionality reduction is a way of simplifying data that has too many variables, keeping only what matters most so it's easier to analyze, visualize, and use in models. It helps computers work faster and more accurately by cutting out redundant or unhelpful information.”
Code Example
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
X = np.random.rand(200, 50) # 50 original features
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=5)
X_reduced = pca.fit_transform(X_scaled)
print("Original shape:", X.shape)
print("Reduced shape:", X_reduced.shape)
print("Variance explained:", pca.explained_variance_ratio_.sum())Follow-up Questions
- What is the difference between feature selection and feature extraction?
- How does PCA determine its principal components?
- Why is standardization important before applying PCA?
- When would you use t-SNE or UMAP instead of PCA?
- What is the curse of dimensionality and how does it affect model performance?
MCQ Practice
1. What is the main goal of dimensionality reduction?
Dimensionality reduction reduces the number of input variables while retaining as much useful information as possible.
2. Which technique projects data onto directions of maximum variance?
PCA finds orthogonal directions (principal components) that capture the most variance in the data.
3. What problem does dimensionality reduction help combat?
As feature count grows, distances become less meaningful and models need exponentially more data — reducing dimensions helps mitigate this curse of dimensionality.
Flash Cards
What is dimensionality reduction? — Reducing the number of features in a dataset while preserving as much meaningful information as possible.
What is the difference between feature selection and feature extraction? — Feature selection picks a subset of existing features; feature extraction creates new combined features, like PCA components.
What does PCA maximize when creating components? — Variance — each principal component captures the maximum remaining variance orthogonal to previous components.
Name a technique used mainly for visualizing high-dimensional data. — t-SNE or UMAP, which create non-linear 2D/3D embeddings for visualization.