What is dimensionality reduction and how does PCA work?
Learn what dimensionality reduction is and how PCA works step by step, using variance, eigenvectors, and principal components to compress data with little loss.
Expected Interview Answer
Dimensionality reduction is the process of reducing the number of input features while retaining as much meaningful information as possible, and PCA (Principal Component Analysis) does this by projecting the data onto a small set of new axes, called principal components, that capture the directions of greatest variance.
High-dimensional data is hard to visualise, slow to train on, and prone to the curse of dimensionality. PCA standardises the features, computes the covariance matrix, and finds its eigenvectors and eigenvalues; the eigenvectors become the principal components and the eigenvalues rank how much variance each explains. By keeping only the top components, PCA compresses correlated features into fewer uncorrelated ones with minimal information loss. It is unsupervised and linear, so it ignores labels and cannot capture non-linear structure.
- Speeds up training and reduces storage
- Mitigates the curse of dimensionality
- Removes redundant, correlated features
- Enables 2D or 3D visualisation of complex data
- Can reduce noise by dropping low-variance components
AI Mentor Explanation
A full match generates thousands of data points, but a highlights reel keeps the few moments that carry most of the story — the wickets and boundaries. PCA is that editor: it finds the directions where the action varies most and keeps them, discarding the many quiet deliveries. You lose little of the narrative while shrinking hours of footage into a compact, information-rich summary.
Step-by-Step Explanation
Step 1
Standardise the data
Centre each feature to zero mean and scale to unit variance so no feature dominates by its units.
Step 2
Compute the covariance matrix
Measure how the features vary together across the dataset.
Step 3
Find eigenvectors and eigenvalues
Decompose the covariance matrix; eigenvectors are the principal component directions, eigenvalues their variance.
Step 4
Rank and select components
Sort components by eigenvalue and keep the top k that explain the desired share of total variance.
Step 5
Project the data
Transform the original data onto the selected components to get the reduced representation.
What Interviewer Expects
- Definition of dimensionality reduction and why it matters
- PCA as variance-maximising linear projection
- Role of covariance, eigenvectors and eigenvalues
- How to choose the number of components via explained variance
- Limitations: linear, unsupervised, needs scaling
Common Mistakes
- Forgetting to standardise features before PCA
- Treating principal components as original features with meaning
- Applying PCA to non-linear data expecting it to capture curvature
- Fitting PCA on the full dataset including the test set, leaking information
- Confusing PCA with feature selection rather than feature extraction
Best Answer (HR Friendly)
“Dimensionality reduction shrinks the number of variables in a dataset while keeping the important information, which makes models faster and data easier to visualise. PCA does this by finding the few directions that capture most of the variation in the data and describing each record using just those.”
Code Example
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X_scaled)
print('Explained variance ratio:', pca.explained_variance_ratio_)
print('Reduced shape:', X_reduced.shape)Follow-up Questions
- How do you decide how many principal components to keep?
- Why is standardising features essential before PCA?
- What are the limitations of PCA on non-linear data, and what alternatives exist?
- How do eigenvalues relate to explained variance?
MCQ Practice
1. What do principal components maximise?
PCA finds orthogonal directions that capture the maximum variance in the data, ordering components by how much they explain.
2. Why should features be standardised before PCA?
Without standardisation, features measured on larger scales contribute disproportionately to variance and bias the components.
3. Which statement about PCA is correct?
PCA is an unsupervised, linear technique that creates new features (components) from combinations of the originals, ignoring labels.
Flash Cards
What is dimensionality reduction? — Reducing the number of features while keeping as much information as possible.
What does PCA maximise? — Variance — it finds the directions capturing the most variation in the data.
What are principal components? — Orthogonal new axes (eigenvectors of the covariance matrix) ranked by explained variance.
Why standardise before PCA? — So features with large scales don't dominate the variance and skew the components.
A key PCA limitation? — It is linear and unsupervised, so it can't capture non-linear structure or use labels.