Practice — Customer Segmentation Pipeline
This exercise integrates every technique from Module 5: K-Means, agglomerative clustering, DBSCAN, PCA visualisation, and Isolation Forest anomaly detection — all applied to a realistic customer analytics dataset. You will build a complete unsupervised ML pipeline, make data-driven decisions about cluster count, identify anomalous customers, and produce a business-ready segment profile report.
Analogy🏏Cricket
🏏 Think of it like cricket: you are a data analyst for an IPL franchise sponsor. You have behavioural data on 500 corporate clients — purchase frequency, spend volume, engagement time, and support ticket rate. Management wants you to identify natural client archetypes (analogous to player archetypes — power hitters vs steadies vs tailenders) and flag unusually behaved accounts for the sales team. No one has pre-labelled the clients. This is a real, unlabelled discovery task.
Setup and Data Generation
python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN
from sklearn.decomposition import PCA
from sklearn.ensemble import IsolationForest
from sklearn.metrics import silhouette_score
from scipy.cluster.hierarchy import dendrogram, linkage
np.random.seed(42)
# Simulate 500 corporate clients
n = 500
df = pd.DataFrame({
'purchase_freq_monthly': np.clip(
np.concatenate([
np.random.poisson(2, 150), # low-frequency buyers
np.random.poisson(8, 200), # mid-frequency
np.random.poisson(20, 150), # high-frequency
]), 0, None),
'avg_order_value_inr': np.concatenate([
np.random.lognormal(9.5, 0.4, 150), # ~13K INR average
np.random.lognormal(11.0, 0.5, 200), # ~60K INR average
np.random.lognormal(12.5, 0.6, 150), # ~270K INR average
]),
'engagement_minutes_wk': np.clip(
np.random.gamma(3, 10, n), 5, 300),
'support_tickets_yr': np.random.poisson(3, n),
})
# Inject 10 anomalous clients
anomaly_idx = np.random.choice(n, 10, replace=False)
df.loc[anomaly_idx, 'avg_order_value_inr'] *= 15 # suspiciously high
df.loc[anomaly_idx, 'support_tickets_yr'] += 50 # excessive complaints
print(df.describe().round(2))
print(f"\nDataset shape: {df.shape}")
Step 1 — Scale and Run the Elbow + Silhouette Sweep
python
# Scale features
scaler = StandardScaler()
X = scaler.fit_transform(df.values)
# Elbow + Silhouette sweep
inertias, sil_scores = [], []
K_range = range(2, 9)
for k in K_range:
km = KMeans(n_clusters=k, n_init=10, random_state=42)
lbl = km.fit_predict(X)
inertias.append(km.inertia_)
sil_scores.append(silhouette_score(X, lbl))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(K_range, inertias, 'bo-')
axes[0].set(xlabel='K', ylabel='Inertia', title='Elbow Method')
for ax, scores, ylabel, title in [
(axes[1], sil_scores, 'Silhouette Score', 'Silhouette Scores')
]:
ax.plot(K_range, scores, 'rs-')
ax.set(xlabel='K', ylabel=ylabel, title=title)
plt.tight_layout(); plt.show()
best_k = K_range[sil_scores.index(max(sil_scores))]
print(f"Best K by silhouette: {best_k} (score={max(sil_scores):.4f})")
Step 2 — Fit K-Means and Profile Segments
python
# Fit best K-Means
km_final = KMeans(n_clusters=3, n_init=10, random_state=42)
df['kmeans_segment'] = km_final.fit_predict(X)
# Profile segments
profile = df.groupby('kmeans_segment').agg({
'purchase_freq_monthly' : 'mean',
'avg_order_value_inr' : 'mean',
'engagement_minutes_wk' : 'mean',
'support_tickets_yr' : 'mean',
'kmeans_segment' : 'count'
}).rename(columns={'kmeans_segment': 'n_customers'})
print("\n=== Segment Profiles (K-Means) ===")
print(profile.round(1))
# Assign business names based on profile
# (order will vary by run — inspect means to label correctly)
segment_names = {
df.groupby('kmeans_segment')['avg_order_value_inr'].mean().idxmin(): 'Budget Buyers',
df.groupby('kmeans_segment')['avg_order_value_inr'].mean().idxmax(): 'Premium Clients',
}
mid_seg = [s for s in df['kmeans_segment'].unique() if s not in segment_names][0]
segment_names[mid_seg] = 'Mid-Tier Regulars'
df['segment_name'] = df['kmeans_segment'].map(segment_names)
print("\nSegment assignment:", segment_names)
Step 3 — Validate with Agglomerative Clustering
python
# Dendrogram on a 60-sample subset for clarity
subset_idx = np.random.choice(len(X), 60, replace=False)
Z = linkage(X[subset_idx], method='ward')
plt.figure(figsize=(14, 4))
dendrogram(Z, leaf_rotation=90, leaf_font_size=8, color_threshold=6)
plt.axhline(6, color='red', ls='--', label='Cut → K≈3')
plt.title('Agglomerative Dendrogram (60-sample subset)')
plt.legend(); plt.tight_layout(); plt.show()
# Full agglomerative on all 500 points
agg = AgglomerativeClustering(n_clusters=3, linkage='ward')
df['agg_segment'] = agg.fit_predict(X)
agg_sil = silhouette_score(X, df['agg_segment'])
km_sil = silhouette_score(X, df['kmeans_segment'])
print(f"K-Means silhouette: {km_sil:.4f}")
print(f"Agglomerative silhouette: {agg_sil:.4f}")
Step 4 — PCA Visualisation
python
# Project to 2D for scatter plot
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)
ev = pca.explained_variance_ratio_
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
colours = ['#E63946', '#2D6A4F', '#457B9D']
for seg, col in zip(df['kmeans_segment'].unique(), colours):
m = df['kmeans_segment'] == seg
name = df.loc[m, 'segment_name'].iloc[0]
axes[0].scatter(X_2d[m, 0], X_2d[m, 1], c=col, label=name, s=20, alpha=0.7)
axes[0].set(xlabel=f'PC1 ({ev[0]*100:.1f}%)', ylabel=f'PC2 ({ev[1]*100:.1f}%)',
title='Customer Segments in PCA Space')
axes[0].legend(fontsize=8)
# Agglomerative on same axes
for seg, col in zip(sorted(df['agg_segment'].unique()), colours):
m = df['agg_segment'] == seg
axes[1].scatter(X_2d[m, 0], X_2d[m, 1], c=col, label=f'Agg {seg}', s=20, alpha=0.7)
axes[1].set(xlabel=f'PC1 ({ev[0]*100:.1f}%)', ylabel=f'PC2 ({ev[1]*100:.1f}%)',
title='Agglomerative Segments in PCA Space')
axes[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
Step 5 — Anomaly Detection with Isolation Forest
python
# Isolation Forest on the same scaled data
iso = IsolationForest(n_estimators=200, contamination=0.02, random_state=42)
df['anomaly_flag'] = iso.fit_predict(X) # -1 = anomaly
df['anomaly_score'] = -iso.score_samples(X) # higher = more anomalous
n_flagged = (df['anomaly_flag'] == -1).sum()
print(f"Anomalies flagged: {n_flagged}")
# How many of the 10 injected anomalies were caught?
injected_caught = (df.loc[anomaly_idx, 'anomaly_flag'] == -1).sum()
print(f"Injected anomalies caught: {injected_caught}/10")
# Top 15 most anomalous customers
print("\nTop 10 highest-scoring anomalies:")
print(df.nlargest(10, 'anomaly_score')[
['purchase_freq_monthly','avg_order_value_inr',
'support_tickets_yr','anomaly_score','segment_name']
].round(2).to_string())
Step 6 — Final Segment Report
python
# Final summary report
print("=" * 60)
print(" CUSTOMER SEGMENTATION REPORT — Sri Hayavadhana Info-Tech")
print("=" * 60)
report = df.groupby('segment_name').agg(
customers = ('kmeans_segment', 'count'),
avg_freq = ('purchase_freq_monthly', 'mean'),
avg_order_inr = ('avg_order_value_inr', 'mean'),
avg_engage_min = ('engagement_minutes_wk', 'mean'),
avg_tickets = ('support_tickets_yr', 'mean'),
anomalies_in_seg= ('anomaly_flag', lambda x: (x==-1).sum())
).round(1)
print(report.to_string())
print(f"\nTotal anomalies flagged : {n_flagged}")
print(f"Clustering silhouette : {km_sil:.4f}")
print(f"Algorithm : K-Means (K=3), validated by Agglomerative")
print("\n✅ Module 5 — Unsupervised Learning complete!")
Challenge Extensions
If you finish early, try these extensions: (1) Run DBSCAN on the scaled data and compare its noise points to the Isolation Forest anomaly flags — how much overlap is there? (2) Add PCA(n_components=0.95) before K-Means in a Pipeline and see if the silhouette score improves. (3) Try K=4 and K=5 — do the additional segments have a plausible business interpretation? (4) Export the final `df` with segment labels to a CSV and write a function that assigns a new customer to a segment using the fitted K-Means model.
Analogy🏏Cricket
🏏 Think of it like cricket: finishing the core drill early is your chance to run optional net sessions that sharpen understanding. Just as you'd cross-check two scouts' verdicts on the misfits — does the density scout (DBSCAN) flag the same odd players the isolation scout does, comparing DBSCAN noise points against Isolation Forest anomaly flags reveals how much the two methods agree. Just as a coach might first strip each player's stats down to their most telling few qualities before grouping them, adding PCA(0.95) ahead of K-Means in a pipeline and watching the silhouette score tests whether cleaner axes give tighter segments. Just as you'd trial four-team versus five-team splits of the squad to see which produces genuinely distinct, coachable roles, testing K=4 against K=5 asks whether extra segments carry real business meaning. And just as you'd file the final team sheet with each player tagged, exporting df with segment labels captures the result. These extensions turn a completed exercise into deeper mastery of when each technique actually earns its place.
- Always scale before any distance-based unsupervised algorithm — K-Means, DBSCAN, agglomerative, and Isolation Forest all depend on feature magnitude.
- Use the Elbow Method and Silhouette Score together to choose K; validate the chosen K with agglomerative clustering's dendrogram.
- PCA 2D projection lets you visually confirm that clusters are well-separated; always report the explained variance percentage.
- Isolation Forest with contamination tuned to the expected anomaly rate is the most practical large-scale anomaly detector.
- Segment profiling (groupby + agg on original unscaled features) is essential — cluster labels alone are meaningless without business interpretation.
- A complete unsupervised pipeline: scale → choose K → cluster → visualise with PCA → detect anomalies → profile and name segments.
- Challenge yourself to compare K-Means and DBSCAN noise on the same dataset — overlap between anomaly methods increases confidence in true outliers.