What is PCA (Principal Component Analysis)?
Understand PCA for dimensionality reduction: how principal components capture variance, why to standardize, choosing components, with scikit-learn code.
Expected Interview Answer
Principal Component Analysis (PCA) is an unsupervised dimensionality-reduction technique that transforms correlated features into a smaller set of uncorrelated axes called principal components, ordered by how much variance they capture. Keeping the top components lets you compress data while retaining most of its information.
PCA finds the directions of maximum variance in the data by computing the eigenvectors and eigenvalues of the feature covariance matrix (equivalently via singular value decomposition). The first principal component points along the greatest variance, each subsequent one is orthogonal to the previous and captures the next-most variance. You standardize the features first, project the data onto the leading components, and choose how many to keep based on cumulative explained-variance. PCA is used for compression, visualization, noise reduction, and speeding up downstream models.
- Reduces dimensionality while preserving most variance
- Removes correlation and multicollinearity between features
- Speeds up training and lowers storage
- Enables 2D/3D visualization of high-dimensional data
- Can reduce noise by dropping low-variance components
AI Mentor Explanation
A selector facing dozens of overlapping stats finds that most player value lines up along a couple of hidden axes, like all-round impact and consistency. PCA does exactly this, collapsing many correlated metrics into a few principal directions that capture most of what distinguishes players, so decisions rest on fewer, more meaningful numbers.
Step-by-Step Explanation
Step 1
Standardize the data
Center each feature to zero mean and scale to unit variance so no feature dominates due to its units.
Step 2
Compute the covariance matrix
Measure how features vary together across the standardized dataset.
Step 3
Find eigenvectors and eigenvalues
Decompose the covariance matrix (or use SVD) to get the principal directions and the variance each explains.
Step 4
Rank and select components
Sort components by eigenvalue and keep the top k that reach the desired cumulative explained variance.
Step 5
Project the data
Transform the original features onto the selected principal components to get the reduced representation.
What Interviewer Expects
- PCA maximizes variance along orthogonal components
- Role of covariance matrix, eigenvectors, and eigenvalues (or SVD)
- Why standardization is required before PCA
- Using explained-variance ratio to choose the number of components
- PCA is unsupervised and produces linear, uncorrelated features
Common Mistakes
- Forgetting to standardize features before applying PCA
- Treating PCA as feature selection rather than feature transformation
- Assuming principal components are always interpretable
- Using PCA on non-linear structure where it underperforms
- Fitting PCA on the full dataset including the test set (leakage)
Best Answer (HR Friendly)
“PCA is a technique that squeezes many overlapping data columns into a few new ones that still capture most of the important variation. It makes large datasets easier to visualize, faster to work with, and less noisy, without losing much information.”
Code Example
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
# Standardize first, then reduce to enough components for 95% variance
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))
X_reduced = pipe.fit_transform(X)
pca = pipe.named_steps['pca']
print('Original features:', X.shape[1])
print('Kept components:', pca.n_components_)
print('Explained variance ratio:', pca.explained_variance_ratio_.round(3))
print('Cumulative:', pca.explained_variance_ratio_.cumsum().round(3))Follow-up Questions
- Why must you standardize features before PCA?
- How do you decide how many principal components to keep?
- What is the relationship between PCA and SVD?
- When does PCA fail, and what non-linear alternatives exist?
- Are principal components interpretable, and why or why not?
MCQ Practice
1. PCA orders its principal components by which criterion?
Principal components are ranked by their eigenvalues, i.e. how much of the data's variance each direction captures, from most to least.
2. Why is standardization important before applying PCA?
PCA is sensitive to feature scale, so without standardization features with larger units dominate the variance and skew the components.
3. Principal components are always:
Each principal component is orthogonal to the others, so the transformed features are mutually uncorrelated by construction.
Flash Cards
What is PCA? — An unsupervised technique that transforms correlated features into fewer uncorrelated principal components ordered by variance explained.
How are principal components found? — Via the eigenvectors/eigenvalues of the covariance matrix, or equivalently through singular value decomposition (SVD).
Why standardize before PCA? — PCA is scale-sensitive; without standardization, features with larger units dominate the variance and distort the components.
How to choose the number of components? — Keep enough top components to reach a target cumulative explained-variance ratio, e.g. 95%.
Are components interpretable? — Not always; each component is a linear mix of original features and may not map cleanly to a single real-world meaning.