Practice — Full EDA on a Housing Dataset

What You'll Build

In this exercise you will build a complete, reusable exploratory data analysis pipeline and apply it end to end to a housing-style dataset framed in cricket terms — analysing factors that predict a player's market valuation. You will implement the full EDA workflow from this module as executable functions: structure inspection, data-quality assessment, univariate profiling for both continuous and categorical variables, bivariate relationship analysis, outlier detection, and a summary of findings with an actionable feature-engineering plan. This consolidates every concept from the module into one practical tool that mirrors exactly how a data scientist opens any new dataset. By the end you will have an EDA engine you can point at any dataset to produce a thorough, documented exploration that surfaces quality issues, distributions, relationships, and anomalies in a single run.

Prerequisites

• Python 3.10 or later with NumPy, SciPy, pandas, and scikit-learn installed.
• Mastery of the EDA workflow and checklist from Lesson 1.
• Command of univariate analysis for continuous and categorical variables from Lesson 2.
• Understanding of bivariate analysis and confounding from Lesson 3.
• Knowledge of outlier detection methods from Lesson 4.

Setup & Project Structure

You will create a project with an EDA engine module containing the reusable analysis functions and a script that runs the engine on the housing dataset. Separating the reusable engine from the dataset-specific script lets you apply the same exploration to any future data. Install the dependencies into a virtual environment and seed any randomness so the analysis, including outlier detection, is reproducible across runs for reliable comparison and debugging.

# Create the project
mkdir cricket_eda_pipeline && cd cricket_eda_pipeline
python -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate
pip install numpy scipy pandas scikit-learn

# Project files
touch eda_engine.py run_eda.py

# Verify
python -c "import numpy, scipy, pandas, sklearn; print('EDA stack ready')"

Step 1 — Foundation

Step 1 builds the foundation: structure-inspection and data-quality functions that report shape, types, missing values, and duplicates. This is the foundation because, per the workflow, understanding structure and quality must precede all analysis, and these functions implement the critical opening checks that catch type errors and missingness before they corrupt anything downstream. Getting this stage right ensures every later stage operates on data the analyst genuinely understands.

# eda_engine.py
import numpy as np
import pandas as pd

def inspect_structure(df):
    """Report shape, dtypes, and a sample - the first checklist step."""
    return {
        "shape": df.shape,
        "dtypes": df.dtypes.astype(str).to_dict(),
        "memory_mb": round(df.memory_usage(deep=True).sum() / 1e6, 2),
    }

def assess_quality(df):
    """Quantify missing values and duplicates - the quality gate."""
    missing = df.isnull().sum()
    return {
        "missing_by_col": missing[missing > 0].to_dict(),
        "missing_pct": (df.isnull().mean() * 100).round(1)[lambda x: x > 0].to_dict(),
        "duplicate_rows": int(df.duplicated().sum()),
    }

if __name__ == "__main__":
    df = pd.read_csv("player_valuation.csv")
    print("Structure:", inspect_structure(df))
    print("Quality:", assess_quality(df))

Step 2 — Core Logic

Step 2 builds the univariate profiling core: a function that profiles every variable with type-appropriate summaries — distributional statistics and skew for continuous variables, frequencies and cardinality for categoricals — and flags quality issues. This is the analytical core because univariate understanding of every variable is the prerequisite for relationship analysis and feature engineering, and producing it uniformly across all variables guarantees nothing is skipped. The function dispatches by type, applying the right summary to each column.

# eda_engine.py (continued)
from scipy import stats

def profile_variable(s):
    """Type-appropriate univariate profile of one column."""
    if s.dtype.kind in "fi":                     # numeric
        clean = s.dropna()
        return {
            "type": "continuous",
            "mean": round(clean.mean(), 2), "median": round(clean.median(), 2),
            "std": round(clean.std(), 2), "skew": round(clean.skew(), 2),
            "range": (round(clean.min(), 2), round(clean.max(), 2)),
        }
    return {                                       # categorical
        "type": "categorical",
        "cardinality": s.nunique(),
        "top": s.value_counts().head(3).to_dict(),
        "rare_pct": round((s.value_counts(normalize=True) < 0.02).mean() * 100, 1),
    }

def profile_all(df):
    return {col: profile_variable(df[col]) for col in df.columns}

Step 3 — Integration & Enhancement

Step 3 integrates bivariate analysis and outlier detection, then assembles everything into a findings summary with an actionable feature-engineering plan. You will add a function that computes the correlation of each numeric feature with the target valuation and flags highly inter-correlated feature pairs, plus IQR-based outlier detection, and a final function that turns all findings into a concrete plan. This integration completes the pipeline by connecting individual-variable understanding to relationships and anomalies, and by converting the entire exploration into the actionable output that makes EDA worthwhile.

# eda_engine.py (continued)
def relationships(df, target):
    """Correlation of numeric features with target, and redundant pairs."""
    num = df.select_dtypes("number")
    target_corr = num.corr()[target].drop(target).round(3).sort_values(key=abs,
                                                                        ascending=False)
    corr_matrix = num.corr().abs()
    redundant = [(a, b) for a in corr_matrix.columns for b in corr_matrix.columns
                 if a < b and corr_matrix.loc[a, b] > 0.85]
    return {"target_correlation": target_corr.to_dict(), "redundant_pairs": redundant}

def detect_outliers(df):
    """IQR-fence outlier counts per numeric column."""
    out = {}
    for col in df.select_dtypes("number").columns:
        s = df[col].dropna()
        q1, q3 = s.quantile([0.25, 0.75]); iqr = q3 - q1
        out[col] = int(((s < q1 - 1.5*iqr) | (s > q3 + 1.5*iqr)).sum())
    return out

def build_plan(df, target):
    """Turn all findings into an actionable feature-engineering plan."""
    plan = {"log_transform": [], "group_rare": [], "drop_redundant": []}
    for col in df.select_dtypes("number").columns:
        s = df[col].dropna()
        if s.skew() > 1.0 and (s > 0).all():
            plan["log_transform"].append(col)
    for col in df.select_dtypes("object").columns:
        if (df[col].value_counts(normalize=True) < 0.02).any():
            plan["group_rare"].append(col)
    plan["drop_redundant"] = relationships(df, target)["redundant_pairs"]
    return plan

Step 4 — Testing & Verification

Now you will run the complete EDA pipeline on the housing valuation dataset and verify the output is coherent and actionable. Confirm the structure and quality checks report sensible shapes and missingness, the univariate profiles correctly classify and summarise each variable, the relationship analysis surfaces the features most correlated with valuation, the outlier detection flags reasonable counts, and the feature-engineering plan lists concrete actions. A complete, sensible report across all stages verifies the pipeline works end to end.

# run_eda.py
import pandas as pd
from eda_engine import (inspect_structure, assess_quality, profile_all,
                        relationships, detect_outliers, build_plan)

df = pd.read_csv("player_valuation.csv")
TARGET = "valuation"

print("=== 1. STRUCTURE ===")
print(inspect_structure(df))
print("\n=== 2. QUALITY ===")
print(assess_quality(df))
print("\n=== 3. UNIVARIATE (sample) ===")
profiles = profile_all(df)
for col in list(profiles)[:3]:
    print(f"  {col}: {profiles[col]}")
print("\n=== 4. RELATIONSHIPS ===")
rel = relationships(df, TARGET)
print(f"  Top correlations with {TARGET}: "
      f"{dict(list(rel['target_correlation'].items())[:3])}")
print(f"  Redundant pairs: {rel['redundant_pairs']}")
print("\n=== 5. OUTLIERS ===")
print(f"  {detect_outliers(df)}")
print("\n=== 6. FEATURE-ENGINEERING PLAN ===")
print(f"  {build_plan(df, TARGET)}")

# Verify: each stage produces sensible, actionable output end to end.

• A reusable EDA pipeline codifies the full workflow into functions applicable to any dataset for fast, consistent exploration.
• Structure and quality inspection must run first, catching type errors and missingness before they corrupt later stages.
• Univariate profiling dispatches type-appropriate summaries to every variable, guaranteeing none is skipped.
• Relationship analysis surfaces features correlated with the target and flags redundant, highly inter-correlated pairs.
• Outlier detection flags anomalies, and combining univariate and multivariate methods strengthens the evidence.
• The pipeline must convert all findings into an actionable feature-engineering plan, the real payoff of EDA.
• Stage order enforces dependencies, so each step relies on the verified output of the previous one.