Armed with the Phase 1 quality report, you now implement the full feature-engineering module for the CricketStream churn dataset. You will fix every quality issue identified in Phase 1, create domain-driven behavioural ratios and aggregations, extract datetime features from the snapshot and signup dates, build targeted interaction features, apply distribution transforms to the skewed columns, and extract TF-IDF features from the support-ticket text. Every transformation respects the leakage rules: all aggregations are shifted, the time-ordered split is enforced before fitting any transformer, and the text vectoriser is fit on training tickets only.
Analogy🏏Cricket
🏏 Think of it like cricket: Before the match, a seasoned captain walks the pitch, examines the surface, studies the weather, and watches the opposition warm up — all observation, no decisions yet. Only after the complete pre-match inspection does he decide on team composition, batting order, and field settings. Just as the captain's observations drive every match-day decision, Phase 1 observations drive every feature-engineering and cleaning decision. Just as skipping the inspection leads to poor decisions based on assumptions, skipping EDA leads to features built on misunderstood data. The insight is that Phase 1 is the pre-match inspection that makes every subsequent decision deliberate rather than accidental.
Step 1 — Clean the Raw Data
Apply the cleaning strategy from the quality report: drop the near-constant column, flag MNAR missingness as an additional binary feature, and split the dataset time-ordered before fitting any imputers. All cleaning transformers are fit on the training split only.
Analogy🏏Cricket
🏏 Think of it like cricket: cleaning the raw data before fitting anything is preparing the pitch and settling the innings order before a single ball is bowled. Just as ground staff remove the dead near-constant patch of worn turf that adds nothing to play, you drop the near-constant column that carries no signal. Just as a suspiciously absent player is itself noted on the team sheet as a meaningful fact, you flag MNAR missingness as its own binary feature rather than silently filling it. Crucially, just as the toss to decide innings order happens before either side sees how the pitch plays — no peeking at the second innings to set up the first — you split the data time-ordered before fitting any imputer, and every cleaning transformer is fit on the training split alone. The payoff: a clean, leakage-free foundation where no future information has bled backward into training, so your later features and model reflect only what you'll actually know at prediction time.
python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
df = pd.read_csv("data/cricketstream_churn.csv", parse_dates=["snapshot_date","signup_date"])
# TIME-ORDERED split (must happen BEFORE any fitting)
df = df.sort_values("snapshot_date")
split_idx = int(len(df) * 0.8)
train = df.iloc[:split_idx].copy()
test = df.iloc[split_idx:].copy()
# Drop near-constant column (identified in Phase 1)
NEAR_CONST_COLS = ["low_variance_col"] # replace with actual column name from Phase 1
train = train.drop(columns=NEAR_CONST_COLS, errors="ignore")
test = test.drop(columns=NEAR_CONST_COLS, errors="ignore")
# MNAR missingness: add binary flag BEFORE imputation (preserves missingness signal)
for col in ["last_ticket_text"]: # fill with high-missing MNAR columns from Phase 1
train[f"{col}_missing"] = train[col].isnull().astype(int)
test[f"{col}_missing"] = test[col].isnull().astype(int)
print(f"Train: {train.shape}, Test: {test.shape}")
print(f"Split date boundary: {train['snapshot_date'].max().date()}")
Step 2 — Domain-Driven Feature Engineering
Create the behavioural ratio and aggregation features motivated by cricket-streaming domain knowledge: engagement efficiency (watch time per session), content breadth (categories per session), plan utilisation (used time vs. plan allowance), and time-aware form features (rolling watch-time trends). All aggregations are shifted to exclude the current snapshot.
Analogy🏏Cricket
🏏 Think of it like cricket: domain-driven feature engineering is turning raw scorecard numbers into the derived stats scouts actually judge players on. Just as strike rate — runs per ball rather than raw runs — reveals true batting efficiency, engagement efficiency divides watch time by session count to expose how much value each visit delivers. Just as a batsman's shot variety measures breadth of skill, content breadth counts categories per session. Just as comparing a player's output to what his role demands shows utilisation, plan utilisation compares used time to the plan's allowance. And just as recent-form indicators — runs across the last five innings — predict the next match better than a career average, time-aware rolling watch-time trends capture a subscriber's momentum. Critically, just as current-match runs are excluded when judging pre-match form, every aggregation is shifted to exclude the current snapshot and avoid leakage. The payoff: features that encode real cricket-streaming behaviour, giving the model the predictive signal raw columns hide.
python
def engineer_domain_features(df):
df = df.copy()
# RATIOS: efficiency signals
df["mins_per_session"] = df["weekly_watch_mins"] / df["total_sessions"].replace(0, np.nan)
df["cats_per_session"] = df["content_categories_watched"] / df["total_sessions"].replace(0, np.nan)
df["plan_utilisation"] = df["weekly_watch_mins"] / df.get("plan_mins_included", 60)
df["notify_click_rate"] = (df["push_notifications_clicked"] /
df["app_opens"].replace(0, np.nan))
# TIME-AWARE aggregation (shift=1 -> no leakage)
df = df.sort_values(["subscriber_id", "snapshot_date"])
df["watch_trend_4w"] = (df.groupby("subscriber_id")["weekly_watch_mins"]
.transform(lambda s: s.shift(1).rolling(4, min_periods=1).mean()))
df["watch_prev_4w"] = (df.groupby("subscriber_id")["weekly_watch_mins"]
.transform(lambda s: s.shift(5).rolling(4, min_periods=1).mean()))
df["engagement_decline"] = df["watch_trend_4w"] - df["watch_prev_4w"]
# INTERACTION: low live-cricket usage on premium plan is a strong churn signal
df["low_live_premium"] = ((df["live_vs_vod_ratio"] < 0.3) &
(df["plan_tier"] == "premium")).astype(int)
return df
train = engineer_domain_features(train)
test = engineer_domain_features(test)
print(f"Features after domain engineering: {train.shape[1]}")
Step 3 — Datetime + Distribution Transform + Text Features
Extract tenure and recency datetime features, log-transform the skewed watch-time columns (fit on training only), and extract TF-IDF features from last_ticket_text (fit on training tickets only, applied to both splits). These complete the feature set motivated by the Phase 1 analysis.
Analogy🏏Cricket
🏏 Think of it like cricket: this step converts three kinds of raw record into model-ready signal, the way a scout translates a player's career into comparable numbers. Just as you convert a debut date into 'years of experience' and 'matches since last game', you extract tenure and recency datetime features from the timestamps. Just as you'd compress a lopsided record — where one giant score dwarfs the rest — onto a fairer scale before comparing players, you log-transform the skewed watch-time columns, fitting the transform on the training split only. And just as you'd read a player's post-match interviews for sentiment but only ever from past games, never a match not yet played, you extract TF-IDF features from last_ticket_text, fitting the vocabulary on training tickets alone before applying it to both splits. The payoff: datetime, distribution and text signals all engineered leakage-free, completing the feature set the Phase 1 analysis called for.
python
from sklearn.preprocessing import PowerTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
import scipy.sparse as sp
def add_datetime_features(df):
df = df.copy()
df["snapshot_date"] = pd.to_datetime(df["snapshot_date"])
df["signup_date"] = pd.to_datetime(df["signup_date"])
df["days_since_signup"] = (df["snapshot_date"] - df["signup_date"]).dt.days
df["snapshot_dow"] = df["snapshot_date"].dt.dayofweek
df["snapshot_month"]= df["snapshot_date"].dt.month
df["dow_sin"] = np.sin(2 * np.pi * df["snapshot_dow"] / 7)
df["dow_cos"] = np.cos(2 * np.pi * df["snapshot_dow"] / 7)
return df
train = add_datetime_features(train)
test = add_datetime_features(test)
# DISTRIBUTION TRANSFORM: fit on train, apply to test
pt = PowerTransformer(method="yeo-johnson")
skewed_cols = ["weekly_watch_mins", "total_sessions"]
train[skewed_cols] = pt.fit_transform(train[skewed_cols].fillna(0))
test[skewed_cols] = pt.transform(test[skewed_cols].fillna(0))
# TEXT FEATURES: fit TF-IDF on TRAIN tickets only
tfidf = TfidfVectorizer(stop_words="english", max_features=30, ngram_range=(1,2))
tfidf.fit(train["last_ticket_text"].fillna("no ticket"))
train_txt = pd.DataFrame(tfidf.transform(train["last_ticket_text"].fillna("no ticket")).toarray(),
columns=[f"txt_{w}" for w in tfidf.get_feature_names_out()],
index=train.index)
test_txt = pd.DataFrame(tfidf.transform(test["last_ticket_text"].fillna("no ticket")).toarray(),
columns=[f"txt_{w}" for w in tfidf.get_feature_names_out()],
index=test.index)
train = pd.concat([train, train_txt], axis=1)
test = pd.concat([test, test_txt], axis=1)
print(f"Final feature count: {train.shape[1]}")
Step 4 — Feature Inventory
Compile a feature inventory table listing every engineered feature, its type, the Phase 1 finding that motivated it, and the leakage-prevention measure applied. This table is part of the Phase 4 written analysis and documents the engineering decisions that drove model performance.
Analogy🏏Cricket
🏏 Think of it like cricket: the feature inventory is the selection dossier justifying why every player made the squad. Just as a professional selection panel documents each pick with its type — specialist batsman, death bowler, all-rounder — the scouting observation that motivated it, and the fitness clearance confirming it is match-legal, your inventory table lists every engineered feature, its type, the Phase 1 finding that motivated it, and the leakage-prevention measure applied. Just as no player enters the squad without a written rationale a coach can defend to the board, no feature enters the model without its motivation and safeguard recorded. And just as that dossier is read out in the post-series review to explain selection decisions, this table feeds directly into the Phase 4 written analysis. The payoff: a documented, defensible record of every engineering decision, tracing each feature from a data finding through to a leakage-safe implementation.
python
feature_inventory = [
("mins_per_session", "ratio", "efficiency signal from domain knowledge", "NA - no time dimension"),
("plan_utilisation", "ratio", "plan usage signal", "NA - point-in-time ratio"),
("engagement_decline", "aggregation", "declining trend = churn signal", "shift(1)+rolling, no current row"),
("low_live_premium", "interaction", "premium plan + low live usage = churn risk", "point-in-time, no leakage"),
("days_since_signup", "datetime", "tenure effects on churn", "snapshot_date - signup_date"),
("weekly_watch_mins", "transform", "highly skewed (Phase 1 skew analysis)", "Yeo-Johnson fit on train only"),
("txt_billing", "text", "billing complaints predict churn (Phase 1)", "TF-IDF fit on train only"),
]
print(f"{'Feature':30} {'Type':12} {'Motivation':45} {'Leakage prevention'}")
print("-"*110)
for row in feature_inventory:
print(f" {row[0]:28} {row[1]:12} {row[2]:45} {row[3]}")
Warning: Every aggregation in Step 2 must use only pre-snapshot information. If your rolling-average window includes the current snapshot, it leaks the current week's watch time into the feature, inflating the training performance of every feature that correlates with watch time. Verify every aggregation by confirming that shift(1) is applied before the rolling window, and double-check the train-test split date boundary to ensure no test-snapshot data was used in any fitting step.
- Phase 2 translates the Phase 1 quality report into concrete cleaning and engineering actions, one finding per decision.
- The time-ordered split must happen before any transformer fitting to prevent leakage from test-snapshot statistics.
- Domain-driven ratios and shifted aggregations are the highest-value features, combining domain knowledge with leakage prevention.
- Distribution transforms (Yeo-Johnson) and TF-IDF are fit on training data only and applied to both splits.
- A feature inventory table documenting every feature's type, motivation, and leakage-prevention measure is part of the deliverable.