Phase 4 — Final Submission and Course Reflection

Phase 4 is the capstone completion: you assemble the final submission package, run the full end-to-end pipeline from raw data to predictions in a single command, write the analytical reflection documenting the feature-engineering decisions that drove performance, and submit. This phase validates that the pipeline is truly reproducible — that a fresh clone of your project can produce the same model from scratch — and that you can articulate why the features you built work, which is the mark of engineering understood rather than engineering copied.

Step 1 — End-to-End Run Verification

Verify that your pipeline can be reproduced end-to-end from a clean environment. Delete all intermediate outputs, reinstall from requirements.txt, and run the single training command. The pipeline should produce the model file and model card from raw data without any manual intervention. This reproducibility check is the acid test of Phase 3 engineering.

# Terminal commands to verify end-to-end reproducibility
# (run from the project root in a fresh virtual environment)

reproducibility_checklist = [
    "1. pip install -r requirements.txt           # install pinned deps",
    "2. python src/train.py                       # run full training pipeline",
    "3. pytest tests/test_churn_pipeline.py -v    # run test suite -> 5 PASSED",
    "4. python -c 'import joblib; p=joblib.load(\"models/churn_pipeline.pkl\"); print(p)'",
    "   # -> prints pipeline repr, confirming the artifact is loadable",
    "",
    "Optional with DVC:",
    "   dvc repro  # reproduces all stages from raw data",
    "   dvc push   # backs up artifacts to remote storage",
]
for step in reproducibility_checklist:
    print(step)

Step 2 — Feature Importance Analysis

Extract and interpret the feature importance from the trained pipeline. Use permutation importance on the test set to rank features by their actual contribution to the model's generalisation, compare with the domain expectations from Phase 2, and write a two-paragraph interpretation. Every feature you engineered should have a cricket-streaming business story — engagement_decline predicts churn because declining usage is the most reliable behavioural signal, plan_utilisation predicts churn because subscribers who underuse their plan feel they are overpaying, and so on.

import numpy as np
import pandas as pd
from sklearn.inspection import permutation_importance

# Extract permutation importance from the test set (unbiased importance measure)
perm = permutation_importance(
    production_pipeline, X_test, y_test,
    n_repeats=20, random_state=42, scoring="roc_auc"
)
feat_imp = pd.DataFrame({
    "feature": X_train.columns,
    "importance_mean": perm.importances_mean,
    "importance_std":  perm.importances_std,
}).sort_values("importance_mean", ascending=False)

print("Top 10 features by permutation importance:")
print(feat_imp.head(10).to_string(index=False))

print("\n--- Feature Story Template ---")
feature_stories = {
    "engagement_decline":  "Subscribers whose watch time is falling have already mentally churned; this is the earliest reliable signal.",
    "plan_utilisation":    "Low plan utilisation means the subscriber is paying for service they do not use, which directly motivates cancellation.",
    "days_since_signup":   "New subscribers churn differently from long-term ones; tenure captures the lifecycle stage that shapes churn risk.",
    "low_live_premium":    "A premium subscriber who rarely watches live cricket is paying for the core product but not using it.",
}
for feat, story in feature_stories.items():
    print(f"  {feat}: {story}")

Step 3 — Written Analytical Reflection

Write a one-to-two page analytical reflection addressing four questions: Which three engineered features contributed most to model performance and why, tracing each to a specific Phase 1 finding? What was the most significant leakage risk you encountered and how did you prevent it? If you were to improve the model further, which additional feature would you engineer and why? What would you do differently if you repeated this project from scratch? The reflection is the deliverable that distinguishes a practitioner who understood the work from one who executed it mechanically.

# Reflection template — fill in with your specific findings

reflection_structure = {
    "top_3_features": {
        "feature_1": {
            "name": "engagement_decline",
            "phase1_finding": "watch-time trend showed consistent decline 4-8 weeks before churn",
            "engineering_decision": "rolling mean with shift(1) to prevent leakage",
            "auc_contribution": "removed -> AUC drops from X to Y (fill from ablation)",
        },
        "feature_2": {"name": "(your second feature)", "phase1_finding": "...",
                       "engineering_decision": "...", "auc_contribution": "..."},
        "feature_3": {"name": "(your third feature)", "phase1_finding": "...",
                       "engineering_decision": "...", "auc_contribution": "..."},
    },
    "leakage_risk": {
        "risk": "engagement_decline rolling mean included current snapshot initially",
        "detection": "CV AUC dropped 0.12 when test AUC was compared",
        "fix": "added shift(1) before rolling window"
    },
    "next_feature": "(describe one additional feature you would build and why)",
    "lessons_learned": "(what you would do differently)",
}

import json
print(json.dumps(reflection_structure, indent=2))
print("\nFill in each field with your specific findings from Phases 1-3.")

Step 4 — Submission Package

Assemble the complete submission package. The deliverables are: the project repository (all source code, tests, and configuration), the model files (churn_pipeline.pkl and churn_pipeline_card.json), the test results (pytest output showing all tests passing), and the written reflection. Zip the project directory and submit. The evaluator will run python src/train.py followed by pytest tests/ -v and expect both to succeed from a clean environment with only the requirements.txt dependencies installed.

# Final submission checklist

submission_checklist = [
    ("src/pipeline.py",              "Production pipeline definition"),
    ("src/train.py",                 "End-to-end training script"),
    ("tests/test_churn_pipeline.py", "pytest test suite (5 tests, all passing)"),
    ("models/churn_pipeline.pkl",    "Fitted pipeline (joblib, compressed)"),
    ("models/churn_pipeline_card.json","Model card (versions, metrics, schema)"),
    ("requirements.txt",             "Pinned dependencies for reproducibility"),
    ("reflection.md",               "Analytical reflection (1-2 pages)"),
    ("dvc.yaml (optional)",          "DVC pipeline for data reproducibility"),
    ("data/cricketstream_churn.csv.dvc (optional)", "DVC-tracked data"),
]
print("Submission checklist:")
for filepath, description in submission_checklist:
    print(f"  [ ] {filepath:45} {description}")
print("\nEvaluator commands:")
print("  pip install -r requirements.txt")
print("  python src/train.py")
print("  pytest tests/ -v")
print("  -> All tests must pass and AUC must match model card.")

Course Reflection

You have completed the Data Analysis and Feature Engineering course. Across 35 lessons and six modules you built the complete toolkit that separates a data scientist who can run a notebook from one who can build a production-ready ML system. You learned to see the patterns hidden in raw data through EDA, to fix the quality issues that corrupt models through systematic cleaning, to create the domain-driven features that unlock signal no raw variable contains, to select the features that matter and discard the noise, and to package everything in a tested, reproducible, deployable pipeline. The capstone project demonstrated that these skills are not separate techniques but a coherent workflow — each phase informing the next, each decision traceable to a specific finding, each feature earning its place by contributing measurable signal. The most important lesson of the course is the one the capstone makes undeniable: in applied data science, good features built with domain knowledge and rigorous leakage prevention consistently outperform sophisticated models built on raw data. A model is only as good as the features it learns from, and the features are only as good as the analyst who builds them.

• The end-to-end reproducibility check — clean environment, single command, full pipeline — is the acid test of professional engineering.
• Permutation importance on the test set, not impurity importance, gives the unbiased feature-value ranking for the written reflection.
• Every top feature must have a cricket-streaming business story connecting its engineering decision to the churn behaviour it captures.
• The leakage risk section of the reflection demonstrates that you actively hunted for leakage rather than assumed it was absent.
• The submission package is a professional deliverable: reproducible, tested, documented, and explainable, not just a number on a leaderboard.
• Features built with domain knowledge and rigorous leakage prevention consistently outperform sophisticated models on raw data.
• The course toolkit — EDA, cleaning, feature engineering, selection, and pipelines — is a coherent workflow, not a collection of isolated techniques.