What You'll Build
In this hands-on exercise you will build a complete data drift monitoring dashboard for IPL cricket player statistics. You will synthesise two datasets — a reference dataset representing last season's player stats and a current dataset representing this season with deliberate distribution shifts in batting_average and innings_count. Using Evidently AI you will generate a DataDriftPreset and TargetDriftPreset report, then build a Streamlit application that displays a drift metrics table, a bar chart of per-feature drift scores, and colour-coded alerts for any feature exceeding a configurable threshold. By the end you will have a reusable monitoring template applicable to any production ML system.
Analogy🏏Cricket
🏏 Think of it like cricket: Before the IPL auction, a franchise's analytics team evaluates every player across dozens of trial matches — tracking Rohit Sharma's strike rate in power plays, Virat Kohli's average against pace, MS Dhoni's finishing rate in the death overs, Shubman Gill's consistency across pitches, and Jasprit Bumrah's economy in the middle overs. Each trial is recorded in a shared logbook so the selectors can compare and pick the best combination. MLflow is exactly that shared logbook for your ML experiments — every training run is a trial match, every metric is a scorecard entry, and the Model Registry is the final squad announcement. Keep the auction framing in mind throughout the exercise, because it fixes the discipline the steps teach: a franchise never signs a player off one good highlight reel, and you never register a model off one lucky run — you log every trial, compare them on identical conditions, and promote only with the full scorecard in front of you.
Prerequisites
Before starting this exercise you should be comfortable with Python 3.9 or later, basic pandas DataFrame operations, and a working understanding of statistical distributions. No prior Evidently AI or Streamlit experience is required — this exercise introduces both libraries from scratch. You need a terminal with a virtual environment where you can install packages freely. Evidently generates HTML and JSON reports entirely offline; no external network access is needed after the pip install step. If you are running inside a Jupyter notebook you can still follow along — replace the Streamlit app section with inline HTML rendering using Evidently's notebook rendering method.
Analogy🏏Cricket
🏏 Think of it like cricket: the kit check before a net session. Just as a batter arriving for practice needs pads, gloves, and a bat they already know how to use — but doesn't need to have faced the new bowling machine before, because today's session is exactly where they'll learn it — this exercise expects you to arrive comfortable with Python, basic scikit-learn (fit, predict, train_test_split), and pandas, while MLflow itself is taught from scratch. Just as the coach insists on a properly prepared practice pitch — a clean, dedicated strip rather than the match square — you need a clean virtual environment or Conda environment where packages can be installed freely. And just as the only outside help needed is the equipment delivery van arriving once before practice, network access is required only for the initial pip install; after that everything runs locally, whether your 'net' is a laptop, a Docker container, or a cloud notebook. The payoff: checking your kit now means the session ahead is pure skill-building, with no stoppages for missing gear.
Setup
python
# Install all required dependencies for the drift monitoring dashboard
# Run this block once before executing any other section
# Option 1: pip (recommended for local environments and cloud notebooks)
!pip install evidently streamlit pandas numpy plotly
# Option 2: conda
# conda install -c conda-forge evidently streamlit pandas numpy plotly
# Verify installations
import evidently
import streamlit
import pandas as pd
import numpy as np
import plotly
print(f"Evidently : {evidently.__version__}")
print(f"Streamlit : {streamlit.__version__}")
print(f"Pandas : {pd.__version__}")
print(f"NumPy : {np.__version__}")
print(f"Plotly : {plotly.__version__}")
print("\nAll dependencies installed successfully!")Step 1: Create Reference and Current Datasets
You will build two pandas DataFrames that simulate IPL player performance across two seasons. The reference dataset reflects last season's stable distributions: batting averages drawn from a normal distribution centred around 38, innings counts between 10 and 16, and strike rates around 132. The current dataset introduces intentional drift: batting averages are shifted downward to a mean of 29 and innings counts are compressed, simulating a harder pitching environment this season. Five legendary players — Rohit Sharma, Virat Kohli, MS Dhoni, Shubman Gill, and Jasprit Bumrah — appear as anchor rows in both datasets with realistic shifted values so you can visually verify the drift before feeding data to Evidently.
Analogy🏏Cricket
🏏 Think of it like cricket: the BCCI maintains two scorebooks — one for the previous IPL season (the reference) and one being filled in right now for the current season (the current window). Virat Kohli's last-season book shows innings_count=15 and batting_average=52; his current-season book shows innings_count=11 and batting_average=38. Jasprit Bumrah's bowling economy has similarly crept up. The selectors do not wait until the end of the season to notice the collapse — they compare the two books mid-season so they can intervene early. Your reference and current DataFrames are those two scorebooks, and Evidently is the analyst who reads them side by side. Two details of the setup deserve attention because they mirror production reality: the drift you inject is *selective* — some columns shift while others stay stable, exactly like real systems where one upstream change moves two features and leaves ten untouched — and the shifts have known ground truth, so you can verify your detector catches what you planted and stays quiet about what you didn't. That planted-fault technique is how real teams validate monitoring before trusting it.
python
import pandas as pd
import numpy as np
# Seed for reproducibility — 18 is Virat Kohli's jersey number
np.random.seed(18)
n = 300 # 300 player-season records in each dataset
# ------------------------------------------------------------------ #
# Reference dataset — last season, stable distributions #
# ------------------------------------------------------------------ #
ipl_reference_df = pd.DataFrame({
"match_id" : [f"IPL2023-{i:04d}" for i in range(n)],
"batting_average" : np.round(np.random.normal(loc=38.0, scale=8.5, size=n).clip(5, 75), 2),
"innings_count" : np.random.randint(10, 17, size=n).astype(float),
"strike_rate" : np.round(np.random.normal(loc=132.0, scale=15.0, size=n).clip(60, 200), 2),
"runs_total" : np.random.randint(180, 900, size=n).astype(float),
"player_label" : np.random.choice(["batsman", "allrounder", "bowler"], size=n,
p=[0.50, 0.30, 0.20]),
})
# Anchor rows: reference season stats for IPL legends
rohit_stats_ref = {"match_id": "IPL2023-ROHIT", "batting_average": 45.2,
"innings_count": 15.0, "strike_rate": 139.5,
"runs_total": 623.0, "player_label": "batsman"}
virat_stats_ref = {"match_id": "IPL2023-VIRAT", "batting_average": 52.8,
"innings_count": 16.0, "strike_rate": 137.2,
"runs_total": 741.0, "player_label": "batsman"}
ipl_reference_df = pd.concat(
[ipl_reference_df, pd.DataFrame([rohit_stats_ref, virat_stats_ref])],
ignore_index=True
)
# ------------------------------------------------------------------ #
# Current dataset — this season, with intentional drift #
# batting_average shifts down (harder pitch conditions) #
# innings_count compressed (more retirements and injuries) #
# ------------------------------------------------------------------ #
ipl_current_df = pd.DataFrame({
"match_id" : [f"IPL2024-{i:04d}" for i in range(n)],
"batting_average" : np.round(np.random.normal(loc=29.0, scale=10.0, size=n).clip(3, 70), 2), # DRIFT: mean drops 38->29
"innings_count" : np.random.randint(7, 13, size=n).astype(float), # DRIFT: range compressed 10-16->7-12
"strike_rate" : np.round(np.random.normal(loc=130.5, scale=15.5, size=n).clip(60, 200), 2), # slight drift
"runs_total" : np.random.randint(120, 750, size=n).astype(float), # mild drift
"player_label" : np.random.choice(["batsman", "allrounder", "bowler"], size=n,
p=[0.45, 0.35, 0.20]), # slight proportion shift
})
# Anchor rows: current season — Rohit and Virat show the drift
rohit_stats_cur = {"match_id": "IPL2024-ROHIT", "batting_average": 31.4,
"innings_count": 11.0, "strike_rate": 118.3,
"runs_total": 378.0, "player_label": "batsman"}
virat_stats_cur = {"match_id": "IPL2024-VIRAT", "batting_average": 38.1,
"innings_count": 12.0, "strike_rate": 129.7,
"runs_total": 512.0, "player_label": "batsman"}
ipl_current_df = pd.concat(
[ipl_current_df, pd.DataFrame([rohit_stats_cur, virat_stats_cur])],
ignore_index=True
)
print("Reference dataset shape:", ipl_reference_df.shape)
print("Current dataset shape:", ipl_current_df.shape)
print("\nReference batting_average stats:")
print(ipl_reference_df["batting_average"].describe())
print("\nCurrent batting_average stats (drift expected — mean should be lower):")
print(ipl_current_df["batting_average"].describe())
print("\nReference innings_count mean:", ipl_reference_df["innings_count"].mean().round(2))
print("Current innings_count mean:", ipl_current_df["innings_count"].mean().round(2))Step 2: Generate Evidently Report
With both datasets prepared, you now run Evidently's DataDriftPreset and TargetDriftPreset to produce a statistical drift report. The DataDriftPreset computes per-feature drift scores using the most appropriate statistical test for each column type — Kolmogorov-Smirnov for continuous features and chi-squared for categorical ones. The TargetDriftPreset checks whether the distribution of the prediction target runs_total has also shifted. You extract the JSON report dictionary to programmatically identify which features are drifted, their drift scores, and their p-values, then pass this structured data to your Streamlit dashboard in the next step.
Analogy🏏Cricket
🏏 Think of it like cricket: the BCCI's official match referee does not simply watch the game and form opinions — he consults a rulebook (the statistical test) and produces a formal written report. If Bumrah's economy in the first six overs drifts above 9.5 for three consecutive matches, the report flags it as a performance anomaly requiring a formal review. The referee uses different tests depending on the metric: for continuous stats like bowling speed he uses a continuous test (Kolmogorov-Smirnov), for categorical outcomes like player role he uses a categorical test (chi-squared). Evidently's DataDriftPreset behaves exactly the same way — it picks the right statistical test automatically for each column type and hands you a clean, structured verdict.
python
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.pipeline.column_mapping import ColumnMapping
import json
# ------------------------------------------------------------------ #
# Build and run the Evidently drift report #
# ------------------------------------------------------------------ #
drift_report = Report(metrics=[
DataDriftPreset(),
TargetDriftPreset(),
])
# ColumnMapping tells Evidently which column is the prediction target
column_mapping = ColumnMapping(
target="runs_total",
numerical_features=["batting_average", "innings_count", "strike_rate"],
categorical_features=["player_label"],
)
drift_report.run(
reference_data=ipl_reference_df.drop(columns=["match_id"]),
current_data=ipl_current_df.drop(columns=["match_id"]),
column_mapping=column_mapping,
)
# Save HTML report for manual inspection in a browser
drift_report.save_html("ipl_drift_report.html")
print("HTML report saved to: ipl_drift_report.html")
# ------------------------------------------------------------------ #
# Extract JSON dict for programmatic access #
# ------------------------------------------------------------------ #
report_json_str = drift_report.json()
report_dict = json.loads(report_json_str)
# Navigate to DataDriftTable results and extract per-feature scores
feature_drift_score = {}
for m in report_dict["metrics"]:
if m["metric"] == "DataDriftTable":
for feature_name, drift_info in m["result"]["drift_by_columns"].items():
feature_drift_score[feature_name] = {
"drifted" : drift_info.get("drift_detected", False),
"drift_score" : round(drift_info.get("drift_score", 0.0), 4),
"p_value" : round(drift_info.get("p_value", 1.0), 4),
"stattest" : drift_info.get("stattest_name", "unknown"),
}
break
# Also check dataset-level drift flag
dataset_drifted = False
for m in report_dict["metrics"]:
if m["metric"] == "DatasetDriftMetric":
dataset_drifted = m["result"].get("dataset_drift", False)
break
print("\n=== Per-Feature Drift Summary ===")
for feat, info in feature_drift_score.items():
status = "DRIFTED" if info["drifted"] else "stable"
print(f" {feat:<22} score={info['drift_score']:.4f} "
f"p={info['p_value']:.4f} test={info['stattest']:<14} [{status}]")
print(f"\nOverall dataset drift detected: {dataset_drifted}")Step 3: Build Streamlit Dashboard
With drift data extracted as a Python dictionary, you now assemble a Streamlit application that presents the results as an interactive dashboard. The app has three visual components: a styled metrics table showing each feature's drift score, p-value, and detected status with colour-coded row highlighting; a Plotly horizontal bar chart ranking features by drift score with a red threshold line at 0.5; and an alert panel that lists every drifted feature alongside a plain-English recommendation. Save the code below to a file named ipl_drift_dashboard.py and launch it with streamlit run ipl_drift_dashboard.py.
Analogy🏏Cricket
🏏 Think of it like cricket: the Mumbai Indians' analytics room has a giant LED scoreboard during the auction — it shows every shortlisted player's recent stats, highlights in red any metric that has fallen below the franchise's minimum threshold, and sounds an alert if a key player like Rohit Sharma or Jasprit Bumrah has drifted so far from last season that the planned bid price is no longer justified. The coaching staff glance at the board between bids and make instant decisions. Your Streamlit dashboard is that auction room scoreboard — colour-coded rows for drifted features, a ranked bar chart so the worst drift is always at the top, and alert cards that translate statistical findings into human-readable action items.
python
# Save this entire block to a file named: ipl_drift_dashboard.py
# Then launch with: streamlit run ipl_drift_dashboard.py
import streamlit as st
import pandas as pd
import numpy as np
import plotly.graph_objects as go
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, TargetDriftPreset
from evidently.pipeline.column_mapping import ColumnMapping
import json
st.set_page_config(page_title="IPL Drift Monitor", page_icon="IPL", layout="wide")
st.title("IPL Player Stats - Drift Monitoring Dashboard")
st.caption("Comparing last season (reference) vs this season (current) distributions")
# ---- Sidebar controls ----
st.sidebar.header("Configuration")
drift_threshold = st.sidebar.slider(
"Drift Score Alert Threshold", min_value=0.1, max_value=1.0,
value=0.5, step=0.05,
help="Features with drift_score above this value are flagged as alerts."
)
n_samples = int(st.sidebar.number_input(
"Synthetic sample size", min_value=100, max_value=2000, value=300, step=100
))
seed_val = int(st.sidebar.number_input("Random seed", value=18))
@st.cache_data
def build_datasets(n, seed):
np.random.seed(seed)
ipl_reference_df = pd.DataFrame({
"batting_average" : np.round(np.random.normal(38.0, 8.5, n).clip(5, 75), 2),
"innings_count" : np.random.randint(10, 17, n).astype(float),
"strike_rate" : np.round(np.random.normal(132.0, 15.0, n).clip(60, 200), 2),
"runs_total" : np.random.randint(180, 900, n).astype(float),
"player_label" : np.random.choice(["batsman","allrounder","bowler"], n, p=[0.50,0.30,0.20]),
})
ipl_current_df = pd.DataFrame({
"batting_average" : np.round(np.random.normal(29.0, 10.0, n).clip(3, 70), 2),
"innings_count" : np.random.randint(7, 13, n).astype(float),
"strike_rate" : np.round(np.random.normal(130.5, 15.5, n).clip(60, 200), 2),
"runs_total" : np.random.randint(120, 750, n).astype(float),
"player_label" : np.random.choice(["batsman","allrounder","bowler"], n, p=[0.45,0.35,0.20]),
})
return ipl_reference_df, ipl_current_df
@st.cache_data
def run_drift_report(n, seed):
ref_df, cur_df = build_datasets(n, seed)
cm = ColumnMapping(
target="runs_total",
numerical_features=["batting_average", "innings_count", "strike_rate"],
categorical_features=["player_label"],
)
report = Report(metrics=[DataDriftPreset(), TargetDriftPreset()])
report.run(reference_data=ref_df, current_data=cur_df, column_mapping=cm)
report_dict = json.loads(report.json())
drift_by_col = {}
for m in report_dict["metrics"]:
if m["metric"] == "DataDriftTable":
for feat, info in m["result"]["drift_by_columns"].items():
drift_by_col[feat] = {
"drifted" : info.get("drift_detected", False),
"drift_score" : round(info.get("drift_score", 0.0), 4),
"p_value" : round(info.get("p_value", 1.0), 4),
"stattest" : info.get("stattest_name", ""),
}
return drift_by_col
with st.spinner("Running Evidently drift analysis on IPL datasets..."):
feature_drift_score = run_drift_report(n_samples, seed_val)
# ---- Section 1: Metrics Table ----
st.subheader("Feature Drift Metrics")
rohit_stats_table = pd.DataFrame([
{
"Feature" : feat,
"Drift Score" : info["drift_score"],
"p-value" : info["p_value"],
"Test" : info["stattest"],
"Status" : "DRIFTED" if info["drifted"] else "Stable",
}
for feat, info in feature_drift_score.items()
]).sort_values("Drift Score", ascending=False)
def colour_status(val):
return ("background-color: #ffcccc; color: #8b0000;"
if val == "DRIFTED" else "background-color: #ccffcc; color: #006400;")
st.dataframe(
rohit_stats_table.style.applymap(colour_status, subset=["Status"]),
use_container_width=True,
)
# ---- Section 2: Bar Chart ----
st.subheader("Drift Score by Feature")
fig = go.Figure()
fig.add_trace(go.Bar(
y=rohit_stats_table["Feature"],
x=rohit_stats_table["Drift Score"],
orientation="h",
marker_color=["#e74c3c" if s == "DRIFTED" else "#2ecc71" for s in rohit_stats_table["Status"]],
text=rohit_stats_table["Drift Score"].round(3),
textposition="outside",
))
fig.add_vline(x=drift_threshold, line_dash="dash", line_color="red",
annotation_text=f"Threshold ({drift_threshold})", annotation_position="top right")
fig.update_layout(
xaxis_title="Drift Score", yaxis_title="Feature",
title="Feature Drift Scores - Red bars exceed threshold, Green bars are stable",
height=350, margin=dict(l=10, r=30, t=50, b=10),
)
st.plotly_chart(fig, use_container_width=True)
# ---- Section 3: Alerts ----
st.subheader("Active Drift Alerts")
drifted_features = [
(feat, info) for feat, info in feature_drift_score.items()
if info["drift_score"] >= drift_threshold
]
if drifted_features:
for feat, info in sorted(drifted_features, key=lambda x: -x[1]["drift_score"]):
st.error(
f"ALERT - {feat} drift score = {info['drift_score']:.4f} "
f"(threshold {drift_threshold}) | p-value = {info['p_value']:.4f} | "
f"Test: {info['stattest']} - consider retraining your production model."
)
else:
st.success("No features exceed the drift threshold. Model input distributions are stable.")
st.caption("Dashboard powered by Evidently AI + Streamlit")Testing and Verification
Run the verification script below to confirm your drift pipeline is working end-to-end without launching the Streamlit UI. The script checks five conditions: both DataFrames have the expected shape, the batting_average mean in the current dataset is at least 5 points lower than the reference confirming synthetic drift, the Evidently report ran and produced a feature_drift_score dict with entries for all four features, at least one feature is flagged as drifted, and the drift score for batting_average specifically exceeds 0.4. All five checks must print PASS before you proceed to launching the Streamlit dashboard.
Analogy🏏Cricket
🏏 Think of it like cricket: verifying a monitoring system is testing the smoke alarm, and the only honest test is lighting a controlled fire. A stadium safety officer does not certify the fire system by admiring the control panel — she triggers a known, contained smoke source in bay 4 and confirms three things: the alarm for bay 4 fires (the planted drift is detected), the alarms for bays 1 through 3 stay silent (the stable features are not falsely flagged), and the control room receives the structured signal it needs to act (your extracted drift dictionary carries the right feature names and scores). Your verification script is precisely this drill: because you *engineered* the drift into specific columns in Step 1, you know exactly which alarms must ring, and a detector that fires everywhere — or nowhere — fails the drill regardless of how handsome the dashboard looks. Carry the habit forward: every production monitoring pipeline deserves a periodic planted-fault test, because the most dangerous monitoring system is not the one that is missing, but the one everybody trusts and nobody has ever seen catch a real fire.
python
# Verification script — run this cell after completing Steps 1 and 2
# All 5 checks must print PASS
passed = 0
failed = 0
def check(label, condition, detail=""):
global passed, failed
status = "PASS" if condition else "FAIL"
if condition:
passed += 1
else:
failed += 1
suffix = f" -> {detail}" if detail else ""
print(f"[{status}] {label}{suffix}")
print("=" * 58)
print(" Drift Dashboard Verification - IPL Season Comparison")
print("=" * 58)
# CHECK 1 - Both DataFrames have the right number of rows and columns
try:
ref_ok = ipl_reference_df.shape[0] >= 300 and ipl_reference_df.shape[1] == 6
cur_ok = ipl_current_df.shape[0] >= 300 and ipl_current_df.shape[1] == 6
check("DataFrames have correct shape", ref_ok and cur_ok,
f"ref={ipl_reference_df.shape}, cur={ipl_current_df.shape}")
except Exception as e:
check("DataFrames have correct shape", False, str(e))
# CHECK 2 - batting_average drift is present (mean drops >= 5 points)
try:
ref_mean = ipl_reference_df["batting_average"].mean()
cur_mean = ipl_current_df["batting_average"].mean()
mean_drop = ref_mean - cur_mean
check("batting_average mean drift >= 5 pts", mean_drop >= 5.0,
f"ref_mean={ref_mean:.2f}, cur_mean={cur_mean:.2f}, drop={mean_drop:.2f}")
except Exception as e:
check("batting_average mean drift >= 5 pts", False, str(e))
# CHECK 3 - feature_drift_score dict has all expected features
try:
expected_features = {"batting_average", "innings_count", "strike_rate", "player_label"}
actual_features = set(feature_drift_score.keys())
check("All 4 features present in drift report",
expected_features.issubset(actual_features),
f"found: {actual_features}")
except Exception as e:
check("All 4 features present in drift report", False, str(e))
# CHECK 4 - At least one feature is flagged as drifted
try:
n_drifted = sum(1 for info in feature_drift_score.values() if info["drifted"])
check("At least 1 feature drifted", n_drifted >= 1,
f"{n_drifted} feature(s) drifted")
except Exception as e:
check("At least 1 feature drifted", False, str(e))
# CHECK 5 - batting_average drift_score > 0.4 (strong drift expected)
try:
ba_score = feature_drift_score.get("batting_average", {}).get("drift_score", 0.0)
check("batting_average drift_score > 0.4", ba_score > 0.4,
f"drift_score={ba_score:.4f}")
except Exception as e:
check("batting_average drift_score > 0.4", False, str(e))
print("=" * 58)
print(f"Result: {passed} passed, {failed} failed")
if failed == 0:
print("ALL CHECKS PASSED - now launch the dashboard:")
print(" streamlit run ipl_drift_dashboard.py")
else:
print("Fix the FAIL items above and re-run this cell.")Warning: Evidently's API changed significantly between v0.2 and v0.4. The DataDriftPreset and TargetDriftPreset shown here require evidently>=0.4.0. If you have an older version installed the Report class will not be importable from evidently.report and the JSON structure will differ. Always pin your version in requirements.txt — for example evidently==0.4.30 — and re-test after upgrades. Additionally, the drift_by_columns key in the JSON output may be absent if no numerical or categorical columns are detected; always pass an explicit ColumnMapping to avoid Evidently guessing column types incorrectly and producing an empty drift summary.
Pro Tip
In production, replace the synthetic datasets with real data fetched from your feature store or data warehouse. Schedule the drift report to run automatically every 24 hours using Airflow or a cron job, then write the feature_drift_score dict to a time-series database like InfluxDB or a simple Postgres table. Your Streamlit dashboard can then query historical drift scores and plot drift trends over time — letting you distinguish whether batting_average drift appeared suddenly (a data pipeline bug) or gradually (genuine concept drift requiring model retraining). Pair the dashboard with PagerDuty or Slack webhook alerts so the on-call engineer is notified immediately when drift_score exceeds the threshold.
- DataDriftPreset automatically selects the statistically correct test per column type — Kolmogorov-Smirnov for continuous features like batting_average and chi-squared for categorical ones like player_label.
- Always provide an explicit ColumnMapping to Evidently specifying numerical, categorical, and target columns; without it Evidently infers types and can misclassify features, producing misleading drift results.
- Extract drift results as JSON with report.json() rather than parsing HTML output, giving you a structured Python dict you can store in a database and query programmatically for automated alerting logic.
- The drift_score ranges from 0 to 1; a common production threshold is 0.5, but tune this for your domain — stricter thresholds of 0.3 catch subtle shifts early while looser thresholds of 0.7 reduce false alerts.
- In Streamlit, use @st.cache_data on both dataset generation and report computation functions so the expensive Evidently run does not repeat on every slider or widget interaction by the user.
- Colour-code your monitoring table and bar chart clearly — red for drifted features, green for stable — so on-call engineers can assess model health at a single glance without reading raw numbers.
- Schedule drift reports to run automatically every 24 hours and store results in a time-series database so you can distinguish sudden data pipeline failures from gradual concept drift requiring model retraining.