What You'll Build
In this exercise you will build a governance and security layer for a cricket model serving system, combining the responsible-AI concerns of Module 6 into one auditable, defended gate. You will produce per-decision explanations with SHAP, compute fairness metrics across a protected group and gate promotion on them, and wrap the serving endpoint with security defences, rate limiting, output coarsening, and input validation, while logging every decision for audit. This pulls together explainability, fairness, and security into the kind of governance harness that regulated production ML actually requires. By the end you will have a system where each prediction comes with a reason, the model is checked for disparate impact before it can be promoted, and the prediction interface resists extraction and adversarial probing, all recorded in an audit trail, the concrete machinery that turns an accurate model into a defensible, accountable, and defended one fit for consequential decisions.
Prerequisites
- Completion of lessons 26 through 29 (explainability, fairness, cost, security), or equivalent familiarity with SHAP, fairness metrics, and ML threats.
- Python 3.11 with the ability to install shap, scikit-learn, numpy, and fastapi.
- A trained tabular classifier (such as the cricket in-form model from earlier modules) and a small labelled evaluation set including a protected-group column.
- Understanding of feature attribution, group fairness metrics, and the model-extraction and adversarial threats from the reading lessons.
- Comfort writing functions and a basic understanding of HTTP request handling for the serving wrapper.
Setup & Project Structure
You will create a project that separates the governance components, explanation, fairness, and security, from the model and the serving layer, the structure a real responsible-AI harness uses. The governance modules live under governance, the serving wrapper under serving, and audit logs under a dedicated directory. Keeping governance concerns modular matters because each, explainability, fairness gating, and security, is independently testable and reusable across models, and an auditor or regulator can inspect each in isolation. Lay out the structure and install the dependencies before building the components.
# Create the governance + security project skeleton.
mkdir -p cricket-governance/{governance,serving,audit}
cd cricket-governance
python -m venv .venv && source .venv/bin/activate
pip install shap==0.45.* scikit-learn==1.* numpy==1.26.* fastapi==0.111.* uvicorn==0.30.*
# Resulting structure:
# cricket-governance/
# |-- governance/
# | |-- explain.py # Step 1: SHAP per-decision explanations
# | |-- fairness.py # Step 2: group metrics + promotion gate
# | `-- security.py # Step 3: rate-limit, coarsen, validate inputs
# |-- serving/
# | `-- serve.py # Step 3: defended endpoint + audit logging
# |-- audit/
# | `-- decisions.log # the audit trail (decision + reason + checks)
# `-- model.pkl # the classifier under governance
echo 'Governance + security project skeleton ready.'
Step 1 — Foundation
Step 1 builds the explainability foundation: a component that attaches a SHAP explanation to every prediction, so no decision leaves the system without a reason. The concept behind this step is accountability, a prediction that cannot be explained cannot be defended to a user or regulator, so the explainer is the first governance layer and the base on which fairness and audit build. You will create an explainer that, for any input, returns the prediction together with the signed feature contributions that drove it, ready to be logged and shown. Getting this foundation right means every recorded decision is interpretable after the fact, the prerequisite for a genuine audit trail.
# governance/explain.py -- Step 1: attach a SHAP explanation to every prediction.
import numpy as np
import shap
FEATURES = ['batting_average', 'strike_rate', 'boundary_pct', 'away_average']
class Explainer:
def __init__(self, model, background=None):
# TreeExplainer is exact + deterministic for tree models.
self.model = model
self.explainer = shap.TreeExplainer(model)
def explain(self, x):
x = np.asarray(x).reshape(1, -1)
pred = int(self.model.predict(x)[0])
shap_vals = np.ravel(self.explainer.shap_values(x))
# Signed contribution per feature -> a defensible, per-decision reason.
contributions = sorted(
({'feature': f, 'contribution': round(float(v), 4)}
for f, v in zip(FEATURES, shap_vals)),
key=lambda c: -abs(c['contribution']))
return {'prediction': pred, 'top_reasons': contributions[:3]}
if __name__ == '__main__':
import pickle
model = pickle.load(open('model.pkl', 'rb'))
ex = Explainer(model)
print(ex.explain([53.6, 131.0, 0.61, 48.0])) # prediction + why
Step 2 — Core Logic
Step 2 builds the core governance logic: a fairness check that computes group metrics across a protected attribute and a promotion gate that blocks a model exhibiting disparate impact. This is the heart of responsible governance because it turns fairness from an aspiration into an enforced precondition, the model cannot be promoted unless its outcomes across protected groups fall within an agreed gap. You will compute selection rate and true-positive rate per group, derive the demographic-parity and equal-opportunity gaps, and implement a gate that fails when a gap exceeds the threshold. This step makes fairness a hard, auditable condition on deployment rather than something hoped for.
# governance/fairness.py -- Step 2: group metrics + a promotion gate.
import numpy as np
def group_metrics(y_true, y_pred, group):
out = {}
for g in np.unique(group):
m = group == g
yt, yp = np.asarray(y_true)[m], np.asarray(y_pred)[m]
pos = yt == 1
out[int(g)] = {
'selection_rate': float(yp.mean()),
'tpr': float(yp[pos].mean()) if pos.any() else 0.0,
}
return out
def fairness_gate(y_true, y_pred, group, max_dp_gap=0.10, max_eo_gap=0.10):
m = group_metrics(y_true, y_pred, group)
sr = [v['selection_rate'] for v in m.values()]
tpr = [v['tpr'] for v in m.values()]
dp_gap = max(sr) - min(sr) # demographic parity gap
eo_gap = max(tpr) - min(tpr) # equal opportunity gap
passed = dp_gap <= max_dp_gap and eo_gap <= max_eo_gap
return {
'passed': passed,
'demographic_parity_gap': round(dp_gap, 3),
'equal_opportunity_gap': round(eo_gap, 3),
'per_group': m,
'verdict': 'PROMOTE' if passed else 'BLOCK: disparate impact exceeds threshold',
}
if __name__ == '__main__':
rng = np.random.default_rng(1983)
n = 1000
group = rng.integers(0, 2, n)
y_true = rng.integers(0, 2, n)
# A model that under-selects qualified members of group 1 -> should be BLOCKED.
y_pred = np.where((group == 1) & (y_true == 1), rng.random(n) < 0.6,
np.where(y_true == 1, rng.random(n) < 0.9, rng.random(n) < 0.1)).astype(int)
print(fairness_gate(y_true, y_pred, group))
Step 3 — Integration & Enhancement
Step 3 integrates the security defences and ties everything into a governed serving endpoint with an audit trail. This integration completes the harness by protecting the prediction interface and recording every decision: the endpoint rate-limits clients and coarsens output to resist extraction, validates inputs to resist adversarial and out-of-distribution probing, and logs each prediction with its explanation for audit. Wrapping the model this way matters because explanation and fairness make the model accountable, but only the security layer keeps it from being stolen or fooled, and only the audit log makes the whole system reviewable after the fact. This step turns the governance components into a single defended, auditable service.
# serving/serve.py + governance/security.py -- Step 3: defended, audited endpoint.
import time, json, os
import numpy as np
from collections import defaultdict, deque
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from governance.explain import Explainer
import pickle
app = FastAPI(title='Governed Cricket Model')
_model = pickle.load(open('model.pkl', 'rb'))
_explainer = Explainer(_model)
_ref = np.load('reference_features.npy') # natural feature ranges
_history = defaultdict(lambda: deque(maxlen=100))
AUDIT_LOG = 'audit/decisions.log'
class Innings(BaseModel):
batting_average: float = Field(..., ge=0, le=120)
strike_rate: float = Field(..., ge=0, le=400)
boundary_pct: float = Field(..., ge=0, le=1)
away_average: float = Field(..., ge=0, le=120)
def rate_limited(client_id, max_qps=5):
now = time.time(); recent = _history[client_id]
while recent and now - recent[0] > 1.0: recent.popleft()
recent.append(now)
return len(recent) > max_qps # anti-extraction
def out_of_distribution(x):
lo, hi = _ref.min(0), _ref.max(0)
return bool(np.any(x < lo - 0.2*(hi-lo)) or np.any(x > hi + 0.2*(hi-lo))) # anti-adversarial
def audit(client_id, x, result):
os.makedirs('audit', exist_ok=True)
with open(AUDIT_LOG, 'a') as f:
f.write(json.dumps({'ts': time.time(), 'client': client_id,
'input': list(x), 'result': result}) + '\n')
@app.post('/predict')
async def predict(f: Innings, client_id: str = 'anon'):
if rate_limited(client_id):
raise HTTPException(status_code=429, detail='rate limit exceeded')
x = np.array([f.batting_average, f.strike_rate, f.boundary_pct, f.away_average])
if out_of_distribution(x):
raise HTTPException(status_code=400, detail='input rejected: out-of-distribution')
explained = _explainer.explain(x) # prediction + reasons
# Output coarsening: return label + reasons, NOT raw probabilities (anti-extraction).
result = {'in_form': bool(explained['prediction']), 'top_reasons': explained['top_reasons']}
audit(client_id, x, result) # every decision is logged
return result
Step 4 — Testing & Verification
Verify all three governance layers: explanations attach to predictions, the fairness gate blocks a biased model, and the security defences reject abuse while logging every decision. Send a valid request and confirm it returns a prediction with reasons, run the fairness gate on a biased prediction set and confirm it blocks, exceed the rate limit and confirm a 429, send an out-of-distribution input and confirm rejection, then inspect the audit log to confirm each decision was recorded. This confirms the whole governance harness works end to end.
# Exercise explanations, the fairness gate, security defences, and the audit trail.
cd cricket-governance && source .venv/bin/activate
# 1) Explanation attached to a prediction:
python -c "import pickle; from governance.explain import Explainer; \
ex=Explainer(pickle.load(open('model.pkl','rb'))); \
print(ex.explain([53.6,131.0,0.61,48.0]))"
# Expected: {'prediction': 1, 'top_reasons': [{'feature': 'batting_average', ...}, ...]}
# 2) Fairness gate blocks a biased model:
python governance/fairness.py
# Expected: {'passed': False, ..., 'verdict': 'BLOCK: disparate impact exceeds threshold'}
# 3) Security: start the server, then exceed the rate limit and send a bad input.
uvicorn serving.serve:app --port 8000 &
for i in $(seq 1 8); do \
curl -s -X POST 'localhost:8000/predict?client_id=attacker' -H 'content-type: application/json' \
-d '{"batting_average":50,"strike_rate":120,"boundary_pct":0.5,"away_average":45}'; done
# Expected: first 5 succeed, then 429 'rate limit exceeded' (anti-extraction)
curl -s -X POST localhost:8000/predict -H 'content-type: application/json' \
-d '{"batting_average":50,"strike_rate":399,"boundary_pct":0.99,"away_average":119}'
# Expected: 400 'input rejected: out-of-distribution' (anti-adversarial)
# 4) Audit trail recorded every served decision:
tail -n 3 audit/decisions.log
# Expected: JSON lines with timestamp, client, input, and result+reasons.
Warning: A common mistake is logging full feature inputs and explanations to an audit trail without considering that those records may contain sensitive personal data. An audit log is itself a data store with privacy and security obligations, an unprotected log of every decision and its inputs can leak exactly the information you were trying to govern. Restrict access to audit logs, retain them only as long as required, and avoid logging sensitive raw attributes where a hashed or redacted reference suffices.
Extension Challenge: Add query-pattern monitoring that flags a client sending a high volume of systematically-varied inputs, the signature of a model-extraction attempt, and raises an alert rather than only rate-limiting. For a harder stretch, add a fairness drift monitor that recomputes the group metrics on a rolling window of audited production decisions and alerts if the equal-opportunity gap grows over time, so fairness is enforced not just at promotion but continuously in production.
- A governance harness wraps a model in explanation, fairness gating, and security so an accurate model becomes defensible, accountable, and defended.
- Attaching a SHAP explanation to every prediction ensures no decision leaves without a defensible, auditable reason.
- A fairness gate computes group metrics and blocks promotion when the demographic-parity or equal-opportunity gap exceeds an agreed threshold, making fairness an enforced precondition.
- Security defences, rate limiting and output coarsening against extraction and input validation against adversarial probing, protect the prediction interface.
- An audit trail logging each decision with its explanation and the checks applied makes the whole system reviewable after the fact.
- Audit logs are themselves sensitive data stores; restrict access, limit retention, and avoid logging raw sensitive attributes unnecessarily.