PCA for Dimensionality Reduction

PCA for Dimensionality Reduction

In Course 3 you met PCA as a pre-processing step inside supervised pipelines — you used it to reduce feature count before feeding data to a classifier or regressor. In this lesson you return to PCA from a purely unsupervised perspective: no labels, no target variable. Here PCA is used for two intrinsically unsupervised tasks — visualisation (projecting high-dimensional data onto 2–3 axes so you can see cluster structure with your eyes) and compression (representing data with fewer numbers while retaining most information). Both tasks are essential companions to the clustering algorithms you learned in Lessons 25–27.

How PCA Works — A Geometric View

PCA finds the directions (principal components) along which the data varies most. The first principal component (PC1) points in the direction of maximum variance in the dataset. The second component (PC2) points in the direction of maximum remaining variance, subject to being perpendicular to PC1. Each subsequent component captures the next highest variance while remaining orthogonal to all previous components. Because real datasets often have correlated features, a few components can capture most of the total variance, letting you safely discard the rest.

Mathematically, PCA computes the eigendecomposition of the covariance matrix (or the singular value decomposition of the data matrix). The eigenvectors are the component directions; the eigenvalues measure how much variance each component explains. Scikit-learn's `PCA` uses SVD internally, which is numerically stable even for wide matrices. You do not need the linear algebra to use PCA effectively — but understanding that components are variance-maximising axes helps you interpret what they represent.

Explained Variance and the Scree Plot

After fitting PCA, `pca.explained_variance_ratio_` gives the fraction of total variance explained by each component. `pca.explained_variance_ratio_.cumsum()` gives the cumulative total. A scree plot (explained variance vs component index) helps you choose how many components to keep: look for the 'elbow' where adding more components gives diminishing returns. A common rule is to keep enough components to explain 90–95% of variance, though the right threshold depends on the application.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_wine

# Load a real multi-dimensional dataset
data = load_wine()
X, y = data.data, data.target           # 178 samples, 13 features, 3 classes

# Scale first — PCA is variance-based, so scale matters
X_scaled = StandardScaler().fit_transform(X)

# Fit PCA to all components
pca_full = PCA()
pca_full.fit(X_scaled)

# Scree plot
evr = pca_full.explained_variance_ratio_
cum_evr = np.cumsum(evr)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].bar(range(1, len(evr)+1), evr, alpha=0.7)
axes[0].set(xlabel='Component', ylabel='Explained Variance Ratio',
             title='Scree Plot')
axes[1].plot(range(1, len(cum_evr)+1), cum_evr, 'bo-')
axes[1].axhline(0.90, color='red', ls='--', label='90% threshold')
axes[1].set(xlabel='Number of Components', ylabel='Cumulative Explained Variance',
             title='Cumulative Variance')
axes[1].legend()
plt.tight_layout(); plt.show()

n_for_90 = np.argmax(cum_evr >= 0.90) + 1
print(f"Components to explain 90% variance: {n_for_90}")
print(f"Variance explained by first 2 PCs : {cum_evr[1]:.3f}")

2D Visualisation of Cluster Structure

The most common unsupervised use of PCA is projecting high-dimensional data to 2D for a scatter plot. Even if K-Means or DBSCAN labels are available, plotting the first two principal components coloured by cluster assignment lets you visually verify that the algorithm found meaningful structure. If the clusters overlap badly in PCA space, they may not be real — they could be artefacts of the algorithm's assumptions rather than genuine groupings in the data.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_wine

data = load_wine()
X_scaled = StandardScaler().fit_transform(data.data)

# K-Means clustering (unsupervised — we don't use y)
km = KMeans(n_clusters=3, n_init=10, random_state=42)
cluster_labels = km.fit_predict(X_scaled)

# Project to 2D with PCA for visualisation
pca2d = PCA(n_components=2)
X_2d = pca2d.fit_transform(X_scaled)

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
colours = ['#E63946', '#457B9D', '#2D6A4F']

# Left: K-Means clusters
for c in range(3):
    m = cluster_labels == c
    axes[0].scatter(X_2d[m, 0], X_2d[m, 1], c=colours[c],
                    label=f'Cluster {c}', s=50, alpha=0.8)
axes[0].set(xlabel=f'PC1 ({pca2d.explained_variance_ratio_[0]*100:.1f}%)',
             ylabel=f'PC2 ({pca2d.explained_variance_ratio_[1]*100:.1f}%)',
             title='K-Means Clusters in PCA Space')
axes[0].legend()

# Right: true wine variety (as reference)
for c, name in enumerate(data.target_names):
    m = data.target == c
    axes[1].scatter(X_2d[m, 0], X_2d[m, 1], c=colours[c], label=name, s=50, alpha=0.8)
axes[1].set(xlabel=f'PC1', ylabel=f'PC2', title='True Wine Varieties (Reference)')
axes[1].legend()

plt.tight_layout(); plt.show()
print(f"2 PCs explain {pca2d.explained_variance_ratio_.sum()*100:.1f}% of variance")

PCA for Compression — Reconstruct and Measure Loss

PCA compression works by projecting the data to a lower-dimensional space, then inverting the projection back to the original feature space. The reconstruction is an approximation — the gap between original and reconstructed values is the information lost by discarding the smaller components. This is measured by reconstruction error (mean squared difference). Compression is useful for storage reduction, denoising, and speeding up downstream algorithms. You choose the number of components by balancing reconstruction quality against the compression ratio.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_wine

data = load_wine()
X = StandardScaler().fit_transform(data.data)   # shape: (178, 13)

print(f"Original shape: {X.shape}")
print(f"{'Components':>12}  {'Explained Var':>14}  {'Reconst. MSE':>13}  {'Compression':>12}")
print("-" * 58)

for n_comp in [1, 2, 3, 5, 8, 13]:
    pca = PCA(n_components=n_comp)
    X_low  = pca.fit_transform(X)          # (178, n_comp)
    X_reco = pca.inverse_transform(X_low)  # (178, 13)
    mse    = np.mean((X - X_reco) ** 2)
    ratio  = X.shape[1] / n_comp
    var    = pca.explained_variance_ratio_.sum()
    print(f"{n_comp:>12}  {var*100:>13.1f}%  {mse:>13.4f}  {ratio:>10.1f}x")

n_components as a Variance Threshold

Instead of specifying an integer component count, you can pass a float between 0 and 1 to `PCA(n_components=0.95)`. Scikit-learn will automatically select the minimum number of components that explain at least 95% of variance. This is the most robust way to use PCA in production pipelines — the component count adapts to the dataset rather than being hardcoded.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_wine
import numpy as np

X = StandardScaler().fit_transform(load_wine().data)

# Auto-select components for 95% variance retention
pca_95 = PCA(n_components=0.95)
X_reduced = pca_95.fit_transform(X)

print(f"Original features  : {X.shape[1]}")
print(f"Reduced components : {pca_95.n_components_}")
print(f"Variance retained  : {pca_95.explained_variance_ratio_.sum()*100:.1f}%")
print(f"Reduced shape      : {X_reduced.shape}")

Link to Course 3 — Supervised vs Unsupervised PCA

In Course 3 (Data Analysis & Feature Engineering), you used PCA inside a supervised pipeline to reduce collinear features before passing them to a regression or classification model. The algorithm was identical but the purpose was different: there you chose components that helped the downstream estimator generalise better, guided by cross-validation scores. Here in unsupervised mode, you choose components purely based on how much variance they retain — there is no target variable to validate against. The key lesson linking both uses: always scale before PCA, always inspect the scree plot, and always embed PCA inside a Pipeline so the same transformation applies consistently to training and future data.

• PCA finds orthogonal axes (principal components) of maximum variance — the first few components capture most of the data's information.
• Always StandardScale before PCA; variance-based decomposition is dominated by high-magnitude features if unscaled.
• The scree plot (explained variance vs component count) reveals the elbow where adding more components gives diminishing returns.
• Use PCA 2D projections to visually verify cluster structure found by K-Means, DBSCAN, or agglomerative clustering.
• Reconstruction via inverse_transform quantifies information loss — choose n_components to balance compression ratio and reconstruction error.
• Pass a float (e.g. 0.95) to PCA(n_components) to auto-select the minimum components explaining that fraction of variance.
• Supervised PCA (Course 3) chooses components to help a downstream model; unsupervised PCA chooses components by variance alone.