CI/CD and Infrastructure Practice — MLOps & Model Deployment

What You'll Build

In this exercise you will build a complete CI/CD pipeline that automatically validates, trains, and gates a cricket model on every change, paired with Infrastructure-as-Code that provisions its serving environment reproducibly. You will define a reproducible DVC pipeline, wrap it in a GitHub Actions workflow that pulls pinned data, reproduces the pipeline, and blocks a merge if model quality regresses, then write a parameterised Terraform module that provisions the serving infrastructure for dev and production from one definition. This pulls together Module 5's CI/CD and IaC into the automated backbone every production ML team runs. By the end you will have a repository where a pull request triggers a quality-gated training run with a posted metrics diff, and where the entire serving environment can be created or torn down with a reviewed plan, the concrete machinery that makes model and infrastructure changes safe, repeatable, and reviewable rather than manual, fragile, and unrepeatable.

Analogy🏏Cricket

🏏 Think of it like cricket: a selection committee does not pick a squad on a hunch; they run structured trial matches under recorded conditions, log every player's scores, compare candidates on identical criteria, and keep the records so a selection can be justified later. Just as the trial matches are your tracked experiment runs, each player's logged scores are your run metrics. Just as the committee compares candidates on the same pitch to be fair, you compare model configurations on the same data. Just as a defensible selection can be reproduced from the records if challenged, your pipeline can reproduce any run from its logged inputs. The insight is that disciplined, recorded trials, not gut feel, are what make both squad selection and model selection defensible.

Prerequisites

Completion of lessons 21 and 22 (CI/CD and Infrastructure as Code), or equivalent familiarity with GitHub Actions and Terraform.
A GitHub repository where you can add workflows and configure branch protection rules.
Familiarity with DVC pipelines (dvc.yaml, dvc repro) from lesson 03 and the ability to install DVC and scikit-learn.
Terraform installed locally, and conceptual understanding of providers, plan, and apply (a cloud account is optional; you can plan without applying).
Comfort reading YAML and HCL configuration and running commands in a terminal.

Setup & Project Structure

You will create a repository that separates the ML pipeline, the CI/CD workflows, and the infrastructure code, the standard layout for a project where both the model and its environment are versioned. The pipeline and its DVC definition live under src and the repo root, the GitHub Actions workflows under .github/workflows, and the Terraform module and per-environment instantiations under infra. Keeping these concerns separate matters because CI gates the model, CD provisions and deploys, and IaC defines the environment, and each evolves on its own cadence while staying version-controlled together. Lay out the structure and gather the tools before writing the pipeline.

Analogy🏏Cricket

🏏 Think of it like cricket: a well-run academy never dumps kit, players, and match records into one heap; it keeps the practice ground, the coaching staff's playbook, and the scorers' logbook in separate, clearly labelled areas so anyone can find what they need and nothing gets mixed up. Just as you separate the data directory, the src pipeline logic, and the mlruns tracking store, the academy separates its pitches, its coaching manuals, and its performance ledgers. Just as keeping these concerns apart makes a project's structure obvious at a glance, a tidy academy lets a new coach walk in and immediately know where drills, plans, and records live. Just as MLflow writes runs to a local store beside, not inside, the code, the scorers keep the logbook outside the coaching manual so results never overwrite strategy. The payoff: a clean layout means a stranger, or you in six months, can pick up the project and understand exactly how data becomes a model.

bash

# Create the CI/CD + IaC project skeleton.
mkdir -p cricket-cicd/{src,ci,.github/workflows,infra/modules/serving,infra/envs/dev,infra/envs/prod}
cd cricket-cicd
git init && dvc init
pip install dvc[s3]==3.* scikit-learn==1.* pandas==2.*

# Resulting structure:
# cricket-cicd/
# |-- src/
# |   |-- featurize.py       # DVC stage 1
# |   `-- train.py           # DVC stage 2 (writes metrics.json)
# |-- dvc.yaml               # the reproducible pipeline definition
# |-- params.yaml            # tunable parameters (tracked by DVC)
# |-- ci/
# |   `-- check_metrics.py   # the quality gate
# |-- .github/workflows/
# |   |-- ml-ci.yaml         # Step 2: validate + gate on PR
# |   `-- ml-cd.yaml         # (extension) deliver on merge
# `-- infra/
#     |-- modules/serving/   # Step 3: reusable Terraform module
#     `-- envs/{dev,prod}/   # thin per-environment instantiations

echo 'CI/CD + IaC project skeleton ready.'

Step 1 — Foundation

Step 1 builds the reproducible DVC pipeline that CI will run, a featurize stage and a train stage that writes a metrics file. The concept behind this step is that the pipeline is the unit of work CI executes, and it must be reproducible and incremental: parameters externalised to params.yaml so DVC detects changes, stages declaring their dependencies and outputs, and a metrics.json the quality gate can read. Getting this foundation right means a clean CI runner can pull the pinned data and reproduce the exact same model and metrics, the prerequisite for any meaningful automated gate. Without a reproducible pipeline, CI runs would be non-deterministic and the gate meaningless.

Analogy🏏Cricket

🏏 Think of it like cricket: before any trial match, the groundsman prepares one agreed pitch and the same set of balls, so every candidate is judged on identical conditions rather than a surface that changes underfoot. Just as the fixed pitch is your versioned dataset, the standard ball is your deterministic feature function. Just as changing the pitch mid-trial would make scores incomparable, a feature function with hidden randomness would make runs incomparable. Just as recording the pitch and ball used lets you recreate the conditions, recording the data version lets you recreate the inputs. The insight is that fair, repeatable comparison demands a fixed, documented starting surface.

python

# dvc.yaml + params.yaml + src/train.py -- Step 1: the reproducible pipeline.
files = '''
# params.yaml -- externalised so DVC detects parameter changes
train:
  max_depth: 4
  random_state: 1983

# dvc.yaml -- the pipeline definition
stages:
  featurize:
    cmd: python src/featurize.py data/innings.csv data/features.csv
    deps: [data/innings.csv, src/featurize.py]
    outs: [data/features.csv]
  train:
    cmd: python src/train.py data/features.csv model.pkl metrics.json
    deps: [data/features.csv, src/train.py]
    params: [train.max_depth, train.random_state]
    outs: [model.pkl]
    metrics: [metrics.json]
'''

# src/train.py -- trains and writes metrics.json for the gate to read.
train_py = '''
import sys, json, pickle, yaml, csv
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

def main(features_path, model_path, metrics_path):
    p = yaml.safe_load(open("params.yaml"))["train"]
    rows = list(csv.DictReader(open(features_path)))
    X = [[float(r["batting_average"]), float(r["wicket_rate"])] for r in rows]
    y = [int(r["in_form"]) for r in rows]
    model = DecisionTreeClassifier(max_depth=p["max_depth"],
                                   random_state=p["random_state"]).fit(X, y)
    acc = accuracy_score(y, model.predict(X))
    pickle.dump(model, open(model_path, "wb"))
    json.dump({"accuracy": round(acc, 4)}, open(metrics_path, "w"))
    print(f"trained: accuracy={acc:.4f}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], sys.argv[3])
'''
# Run locally with: dvc repro   (reruns only changed stages)
print('Reproducible, incremental pipeline writing metrics.json for the CI gate.')

Step 2 — Core Logic

Step 2 builds the core CI logic: the quality gate and the GitHub Actions workflow that runs the pipeline and enforces it on every pull request. This is the heart of the exercise because it turns the reproducible pipeline into an automatic guardrail, the workflow pulls pinned data, reproduces the pipeline, and runs a gate that fails the build, blocking the merge, if accuracy regresses. The gate is implemented as a process exit code, and branch protection makes a passing run required to merge, which is the mechanism that actually stops a regressing model. This step is what makes model changes as safe to merge as code changes.

Analogy🏏Cricket

🏏 Think of it like cricket: when a candidate bats in a trial, an official scorer records the conditions they batted in, the pitch, the bowling, and their exact score, all under one innings entry. Just as the recorded conditions are your logged params, the score is your logged metric, and the innings entry is the MLflow run. Just as the same conditions and the same player should reproduce a comparable knock, the same params and seed reproduce the same accuracy. Just as a scorer who noted only the score but not the conditions leaves selectors guessing, logging a metric without its params leaves you unable to explain a result. The insight is that a result is only useful when recorded together with the conditions that produced it.

python

# ci/check_metrics.py + .github/workflows/ml-ci.yaml -- Step 2: the gate + CI.
check_metrics = '''
import json, sys, argparse
ap = argparse.ArgumentParser()
ap.add_argument("--min-accuracy", type=float, required=True)
args = ap.parse_args()
acc = json.load(open("metrics.json"))["accuracy"]
if acc < args.min_accuracy:
    print(f"GATE FAILED: accuracy {acc:.4f} < required {args.min_accuracy}")
    sys.exit(1)            # non-zero exit fails the job -> blocks the merge
print(f"GATE PASSED: accuracy {acc:.4f} >= {args.min_accuracy}")
'''

ci_workflow = '''
name: ML CI
on:
  pull_request:
    branches: [main]
jobs:
  train-and-validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r requirements.txt
      - name: Pull pinned data
        run: dvc pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      - name: Reproduce the pipeline
        run: dvc repro
      - name: Quality gate
        run: python ci/check_metrics.py --min-accuracy 0.80
      - name: Post metrics diff on the PR
        run: dvc metrics diff main --md >> $GITHUB_STEP_SUMMARY
'''
# Then enable branch protection on main: require the 'train-and-validate' check to pass.
print('CI pulls pinned data, reproduces the pipeline, and gates the merge on accuracy.')

Step 3 — Integration & Enhancement

Step 3 brings in Infrastructure as Code: a parameterised Terraform module for the serving environment, instantiated for dev and production from one definition. This integration completes the backbone by pairing the gated model pipeline with reproducible infrastructure, the module declares the serving resources once, and dev and prod differ only in variables like instance size and replica count. Defining infrastructure as a reusable module matters because it keeps environments consistent and changes reviewable: a terraform plan shows exactly what a change will create or alter before apply, and the same module guarantees dev and prod share shape. This step makes the environment as version-controlled and reproducible as the model.

Analogy🏏Cricket

🏏 Think of it like cricket: after a round of trial matches, the committee lays every candidate's recorded scores side by side, picks the top performer on the agreed metric, and formally names them to the squad while keeping the rest on record. Just as laying out all scorecards is your sweep across configs, choosing the highest scorer is your selection by logged accuracy. Just as the named player is promoted while others remain in the pool for recall, the best model is promoted while others stay tracked. Just as the selection memo cites the scores that justified it, your promotion cites the winning run ID. The insight is that evidence-based selection across recorded candidates produces a defensible, reversible choice.

python

# Terraform module + per-env instantiations -- Step 3: reproducible serving infra.
tf = '''
# infra/modules/serving/main.tf -- the reusable serving stack (cloud-agnostic shape)
variable "environment"    { type = string }
variable "instance_type"  { type = string }
variable "replica_count"  { type = number }
variable "model_uri"      { type = string }

resource "aws_sagemaker_model" "form" {
  name               = "cricket-form-${var.environment}"
  execution_role_arn = var.exec_role_arn
  primary_container { image = var.image_uri  model_data_url = var.model_uri }
}
resource "aws_sagemaker_endpoint_configuration" "form" {
  name = "cricket-form-cfg-${var.environment}"
  production_variants {
    variant_name           = "primary"
    model_name             = aws_sagemaker_model.form.name
    initial_instance_count = var.replica_count
    instance_type          = var.instance_type
  }
}
resource "aws_sagemaker_endpoint" "form" {
  name                 = "cricket-form-${var.environment}"
  endpoint_config_name = aws_sagemaker_endpoint_configuration.form.name
}

# infra/envs/prod/main.tf -- thin instantiation (prod parameters)
module "serving" {
  source         = "../../modules/serving"
  environment    = "prod"
  instance_type  = "ml.m5.xlarge"
  replica_count  = 3
  model_uri      = "s3://cricket-models/form/v3/model.tar.gz"
}
# infra/envs/dev/main.tf -- SAME module, dev parameters (ml.t2.medium, replica_count=1)
'''

# Review-then-apply workflow (and run plan in CI on infra PRs):
#   cd infra/envs/prod && terraform init
#   terraform plan -out=tf.plan     # preview the exact diff
#   terraform apply tf.plan         # apply only after review
print('One serving module; dev and prod differ only in variables; plan before apply.')

Step 4 — Testing & Verification

Verify the whole backbone: the pipeline reproduces and the gate behaves correctly, the CI workflow blocks a regressing change, and the Terraform module plans cleanly for both environments. Run the pipeline locally and confirm the gate passes above threshold and fails below it, open a pull request with a deliberately weakened model and confirm CI blocks the merge, then run terraform plan for dev and prod and confirm both produce a valid diff from the same module. This confirms the model gate and the infrastructure definition both work end to end.

Analogy🏏Cricket

🏏 Think of it like cricket: before trusting a match plan you verify two things, that it actually works start to finish under real conditions and produces a clear winner, and that it is repeatable, run the same trial again in the same conditions and you get the same result rather than a fluke. Just as you confirm the pipeline runs end to end and produces a tracked leaderboard plus a promoted model, a coach confirms the full session runs from warm-up to selection and yields a ranked shortlist with a clear pick. Just as you rerun the same configuration and check it yields the same accuracy, a coach reruns the identical trial and expects the same player to top the table, proving it was skill, not chance. Just as the MLflow UI records every run with its params, metric, and artifact, the scorebook records every trial so the result can be inspected and defended. The payoff: verifying both correctness and reproducibility, in a pipeline or a trial, means you trust the winner because it holds up when repeated.

bash

# Exercise the pipeline, the gate, the CI block, and the Terraform plan.
cd cricket-cicd

# 1) Reproduce the pipeline and run the gate locally.
dvc repro
python ci/check_metrics.py --min-accuracy 0.80
# Expected: trained: accuracy=0.8xxx  then  GATE PASSED: accuracy 0.8xxx >= 0.8

# 2) Simulate a regression: lower the bar the model must clear, then raise the gate.
python ci/check_metrics.py --min-accuracy 0.99
# Expected: GATE FAILED: accuracy 0.8xxx < required 0.99   (exit code 1)
echo "exit code: $?"   # -> 1, which would fail the CI job and block the merge

# 3) Verify the CI gate blocks a PR: push a branch that regresses the model,
#    open a PR to main, and confirm the 'train-and-validate' check fails red
#    and the merge button is blocked by branch protection.

# 4) Verify the IaC module plans for both environments from one definition.
cd infra/envs/dev  && terraform init && terraform plan   # dev: small instance, 1 replica
cd ../prod         && terraform init && terraform plan   # prod: large instance, 3 replicas
# Expected: both produce a valid plan; the diff differs only in instance_type/replica_count.

Warning: A common mistake is writing the CI workflow but never enabling branch protection to require the check. The workflow then runs and goes red on a regression, but the merge button stays enabled, so a developer can merge the failing change anyway and the gate accomplishes nothing. The exit-code gate only protects the branch when branch protection makes the passing check required; configure that rule, or your CI is advisory, not enforcing.

Extension Challenge: Add a CD workflow that, on merge to main, registers the validated model and transitions it to a Staging stage, then add a manual approval environment gate in GitHub Actions before a Production deploy. For a harder stretch, run terraform plan automatically in CI on any pull request touching infra/ and post the plan as a PR comment, so infrastructure changes are reviewed with the same diff-and-approve discipline as the model.

A CI/CD backbone for ML pairs a quality-gated, reproducible training pipeline with Infrastructure-as-Code that provisions the serving environment.
A reproducible DVC pipeline with externalised params and a metrics output is the unit of work CI runs, so a clean runner reproduces the same model and metrics.
The quality gate is a step whose non-zero exit fails the build; branch protection requiring that check is what actually blocks a regressing merge.
CI must pull pinned data on a clean runner and reproduce the pipeline so results are deterministic and not dependent on local state.
A parameterised Terraform module lets dev and production share one definition and differ only in variables, kept consistent and reviewable via plan before apply.
Run terraform plan in CI on infra changes and review the diff, treating infrastructure changes with the same rigour as model and code changes.