MLOps Fundamentals Cheat Sheet
Summarizes MLOps practices for experiment tracking, model versioning, and continuous training pipelines using tools like MLflow, DVC, and CI/CD automation.
2 PagesIntermediateFeb 28, 2026
Experiment Tracking with MLflow
Log parameters, metrics, and models for every training run.
python
import mlflowimport mlflow.sklearnmlflow.set_experiment("churn-model")with mlflow.start_run(): model.fit(X_train, y_train) acc = model.score(X_test, y_test) mlflow.log_param("n_estimators", 100) mlflow.log_metric("accuracy", acc) mlflow.sklearn.log_model(model, "model")# View the UI: mlflow ui --port 5000
CI Pipeline for Retraining
Automate scheduled model retraining with GitHub Actions.
yaml
name: retrain-modelon: schedule: - cron: "0 3 * * 1" # every Monday 3am workflow_dispatch: {}jobs: train: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.11" - run: pip install -r requirements.txt - run: python train.py - run: python evaluate.py --min-accuracy 0.85
Core Concepts
Foundational ideas behind an MLOps workflow.
- MLOps- Practices that apply DevOps principles (CI/CD, automation, monitoring) to the machine learning lifecycle
- Feature store- Central repository for versioned, reusable features shared between training and serving
- Model registry- Versioned catalog of trained models with lifecycle stages (staging, production, archived)
- Reproducibility- Ability to recreate a model exactly via pinned data, code, and dependency versions
- Data versioning- Tracking dataset versions (e.g. with DVC) alongside code so experiments are traceable
- Continuous training (CT)- Automatically retraining models on new data on a schedule or trigger
Common Tooling
Widely used tools across the MLOps stack.
- MLflow- Open-source platform for experiment tracking, model packaging, and a model registry
- DVC- Data Version Control; git-like versioning for datasets and ML pipelines
- Kubeflow- Kubernetes-native platform for orchestrating ML pipelines
- Airflow- Workflow orchestrator commonly used to schedule ETL and training DAGs
- Weights & Biases- Experiment tracking and visualization tool similar to MLflow
Point-in-Time Correct Feature Retrieval (Feast)
Avoid label leakage by joining historical features as of each label's timestamp, not the latest value.
python
from feast import FeatureStoreimport pandas as pdstore = FeatureStore(repo_path=".")entity_df = pd.DataFrame({ "customer_id": [1001, 1002, 1003], "event_timestamp": pd.to_datetime([ "2026-01-15", "2026-02-01", "2026-02-20" ]),})training_df = store.get_historical_features( entity_df=entity_df, features=[ "customer_stats:avg_order_value", "customer_stats:days_since_last_order", ],).to_df()# At serving time, fetch only the latest values (low latency online store)online_features = store.get_online_features( features=["customer_stats:avg_order_value"], entity_rows=[{"customer_id": 1001}],).to_dict()
Reproducible Pipeline DAG with DVC
Declare training stages so DVC only reruns steps whose inputs actually changed.
yaml
# dvc.yamlstages: prepare: cmd: python prepare.py --input data/raw.csv --output data/processed.csv deps: - data/raw.csv - prepare.py outs: - data/processed.csv train: cmd: python train.py --data data/processed.csv --out model.pkl deps: - data/processed.csv - train.py params: - train.n_estimators - train.max_depth outs: - model.pkl metrics: - metrics.json: cache: false# Run: dvc repro# Compare: dvc metrics diff main --targets metrics.json
Champion/Challenger Evaluation Gate in CI
Block promotion to production unless the challenger model beats the current champion on a held-out evaluation set.
python
import mlflowfrom mlflow.tracking import MlflowClientclient = MlflowClient()champion = client.get_model_version_by_alias("churn-model", "champion")champion_model = mlflow.pyfunc.load_model(f"models:/churn-model/{champion.version}")challenger_acc = evaluate(challenger_model, X_holdout, y_holdout)champion_acc = evaluate(champion_model, X_holdout, y_holdout)if challenger_acc <= champion_acc + 0.005: raise SystemExit( f"Challenger ({challenger_acc:.4f}) did not beat champion " f"({champion_acc:.4f}) by the required margin - blocking promotion" )client.set_registered_model_alias("churn-model", "champion", challenger_version)
Population Stability Index for Drift Alerts
Quantify how much a live feature's distribution has shifted from the training baseline.
python
import numpy as npdef psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float: breakpoints = np.quantile(expected, np.linspace(0, 1, bins + 1)) breakpoints[0], breakpoints[-1] = -np.inf, np.inf e_pct = np.histogram(expected, breakpoints)[0] / len(expected) a_pct = np.histogram(actual, breakpoints)[0] / len(actual) e_pct = np.clip(e_pct, 1e-4, None) a_pct = np.clip(a_pct, 1e-4, None) return float(np.sum((a_pct - e_pct) * np.log(a_pct / e_pct)))# PSI < 0.1: no significant shift# 0.1 <= PSI < 0.25: moderate shift, investigate# PSI >= 0.25: major shift, retrain trigger
MLOps Maturity Levels
Where a team sits on the automation spectrum, from manual to fully automated.
- Level 0 - Manual- Data scientists hand off notebooks; deployment is a manual, ad-hoc process with no pipeline or tracking
- Level 1 - ML pipeline automation- Training is a repeatable pipeline that can be re-run on new data, but deployment to production is still manual
- Level 2 - CI/CD pipeline automation- Source control, automated testing, and CI/CD build/deploy the pipeline itself, not just the model
- Continuous training (CT) trigger- Retraining fires automatically on a schedule, on new data arrival, or on a drift alert rather than manually
- Model lineage- Full traceability from a deployed model back to the exact data version, code commit, and hyperparameters that produced it
- Automated rollback- Monitoring automatically reverts to the previous champion model when live metrics degrade past a threshold, no human in the loop
- Feature/training skew check- CI gate that fails the pipeline if online feature computation logic diverges from the offline training-time computation
Pro Tip
Treat your training pipeline as code: pin dependency versions, seed random states, and version the training data - otherwise 'reproduce this model' becomes impossible six months later.
Was this cheat sheet helpful?
Explore Topics
#MLOpsFundamentals#MLOpsFundamentalsCheatSheet#DataScience#Intermediate#ExperimentTrackingWithMLflow#CIPipelineForRetraining#CoreConcepts#CommonTooling#MachineLearning#DevOps#CheatSheet#SkillVeris
