Practice — Build a Feature Selection Pipeline

What You'll Build

In this exercise you will build a complete, principled feature-selection pipeline that chains filter, wrapper, and embedded methods in the correct order, evaluating each stage's contribution and producing a lean, predictive feature set for a player-performance prediction task. You will implement variance thresholding, correlation filtering, RFECV, and Lasso-based embedded selection, comparing their outputs and combining them into a two-stage filter-then-wrapper pipeline inside a cross-validation loop that prevents feature-selection leakage throughout. This consolidates the entire feature-selection module into one tool that mirrors professional practice — not any single method but a principled chain of complementary methods that together deliver the right features efficiently and honestly.

Prerequisites

• Python 3.10 or later with NumPy, pandas, and scikit-learn installed.
• Mastery of filter methods from Lesson 19 and wrapper methods from Lesson 20.
• Understanding of embedded methods (Lasso and tree importance) from Lesson 21.
• Familiarity with PCA from Lesson 22 and with pipeline-based leakage prevention throughout.
• Comfort with scikit-learn pipelines, GridSearchCV, and cross-validation.

Setup & Project Structure

You will create a project with a selection-pipeline module and a script that applies the full pipeline to a player-performance dataset. Separating the reusable pipeline from the data-specific script lets you apply the same selection to any future high-dimensional dataset. All selectors will be placed inside pipelines to prevent feature-selection leakage during cross-validation, the non-negotiable discipline that makes the evaluation honest.

# Create the project
mkdir cricket_feature_selection && cd cricket_feature_selection
python -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate
pip install numpy pandas scikit-learn

# Project files
touch selection_pipeline.py run_selection.py

# Verify
python -c "import numpy, pandas, sklearn; print('Feature selection stack ready')"

Step 1 — Foundation

Step 1 builds the foundation: the fast, model-free filter stage that removes the obviously irrelevant before any expensive model fitting. You will implement a variance threshold combined with an ANOVA F-score filter, chained together as the first stage of the pipeline. This is the foundation because it is the cheapest selection — requiring no model training — and it narrows the feature space to a manageable size that makes the subsequent wrapper stage computationally feasible.

# selection_pipeline.py
from sklearn.feature_selection import VarianceThreshold, SelectKBest, f_classif
from sklearn.pipeline import Pipeline

def build_filter_stage(k_best=20):
    """Two-step filter: variance threshold (cheap) then F-score (target-aware)."""
    return Pipeline([
        ("vt", VarianceThreshold(threshold=0.01)),      # remove near-constants
        ("kbest", SelectKBest(f_classif, k=k_best)),    # keep top-k by ANOVA F-score
    ])

if __name__ == "__main__":
    import numpy as np
    rng = np.random.default_rng(42)
    X = np.column_stack([np.ones((300,2))*0.01, rng.normal(0,1,(300,28))])
    y = (X[:,2:7].sum(1)>0).astype(int)
    filt = build_filter_stage(k_best=20)
    filt.fit(X, y)
    print(f"After filter stage: {filt.transform(X).shape[1]} features from {X.shape[1]}")

Step 2 — Core Logic

Step 2 builds the core RFECV wrapper stage that follows the filter and precisely identifies the optimal feature subset by cross-validated model performance. This is the analytical core because it uses the model itself to evaluate which features genuinely help prediction, catching the interactions and complementarity that the filter misses, and RFECV automatically determines the right number of features rather than requiring arbitrary specification.

# selection_pipeline.py (continued)
from sklearn.feature_selection import RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold

def build_wrapper_stage():
    """RFECV: let cross-validation find the optimal feature count automatically."""
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    return RFECV(
        estimator=LogisticRegression(max_iter=500, random_state=42),
        cv=cv,
        scoring="roc_auc",
        min_features_to_select=1,
        step=1,
    )

def build_selection_pipeline(k_best=20):
    """Full leakage-safe selection pipeline: filter -> RFECV -> model."""
    return Pipeline([
        ("filter", build_filter_stage(k_best=k_best)),
        ("rfecv", build_wrapper_stage()),
        ("model", LogisticRegression(max_iter=500, random_state=42)),
    ])

Step 3 — Integration & Enhancement

Step 3 integrates Lasso-based embedded selection as an alternative path and adds a comparison function that evaluates all three approaches, assembling the diagnostic view of which method delivered the best features. You will tune the filter's k parameter via GridSearchCV, compare filter, embedded, and filter-then-RFECV approaches by cross-validated AUC, and report which feature sets each method identified. This integration completes the pipeline by connecting all three selection families and providing the comparative view that guides the final choice.

# selection_pipeline.py (continued)
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV
from sklearn.model_selection import cross_val_score, GridSearchCV

def compare_methods(X, y):
    """Compare filter, embedded (Lasso), and filter+RFECV pipeline AUC."""
    results = {}

    # FILTER only (tune k via GridSearchCV)
    filter_pipe = Pipeline([
        ("filter", build_filter_stage(k_best=5)),
        ("model", LogisticRegression(max_iter=500)),
    ])
    gscv = GridSearchCV(filter_pipe, {"filter__kbest__k": [5,10,15,20]},
                        cv=5, scoring="roc_auc")
    gscv.fit(X, y)
    results["filter (best k)"] = (gscv.best_score_, gscv.best_params_)

    # EMBEDDED: Lasso selection
    emb_pipe = Pipeline([
        ("select", SelectFromModel(LassoCV(cv=5), threshold=1e-4)),
        ("model", LogisticRegression(max_iter=500)),
    ])
    results["embedded (Lasso)"] = (cross_val_score(emb_pipe,X,y,cv=5,scoring="roc_auc").mean(), {})

    # FILTER + RFECV (the full pipeline)
    full_pipe = build_selection_pipeline(k_best=20)
    results["filter + RFECV"] = (cross_val_score(full_pipe,X,y,cv=5,scoring="roc_auc").mean(), {})

    return results

Step 4 — Testing & Verification

Now you will run the complete comparison on a controlled dataset where you know the ground truth — exactly which features are relevant — and verify that each method recovers the relevant features and that the full filter-then-RFECV pipeline achieves the best AUC. Comparing selected features against the known relevant set verifies the pipeline encodes the module's logic correctly, and the leakage-free evaluation verifies the pipeline is trusted.

# run_selection.py
import numpy as np
from selection_pipeline import compare_methods, build_selection_pipeline
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
# Known ground truth: first 5 of 50 features are truly relevant
X = rng.normal(0, 1, (600, 50))
y = (X[:, :5].sum(axis=1) + rng.normal(0, 0.5, 600) > 0).astype(int)

print("=== Method Comparison (AUC, higher is better) ===")
results = compare_methods(X, y)
for method, (auc, params) in results.items():
    print(f"  {method:25}: AUC = {auc:.3f}  {params}")

# Verify the full pipeline selects features close to the known relevant set
pipe = build_selection_pipeline(k_best=20)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=42)
pipe.fit(Xtr, ytr)

# Which original features survived the filter stage?
filt_mask = pipe["filter"].get_support()
print(f"\nFilter kept features: {filt_mask.nonzero()[0][:10]} ...")
print("Expected: features 0-4 should be among the survivors.")
print("All selection inside pipeline -> no leakage in the AUC estimates above.")

• A principled feature-selection pipeline chains complementary methods: fast filters for breadth, then wrappers or embedded methods for precision.
• Variance thresholding and F-score filtering remove the obviously irrelevant cheaply, making the expensive wrapper stage computationally feasible.
• RFECV automatically determines the optimal feature count via cross-validated model performance, avoiding arbitrary specification.
• Lasso embedded selection provides an efficient alternative path that performs selection and fitting in one model-fitting step.
• Comparing filter, embedded, and filter-then-wrapper AUC with cross-validation guides the method choice for each dataset.
• Every selector must be inside the pipeline so it refits on each training fold; selectors fit outside the pipeline leak test information.
• Tune the filter's k parameter via GridSearchCV inside the pipeline, treating it as a hyperparameter to be optimised.