Kubeflow Cheat Sheet
Run ML pipelines, hyperparameter tuning, and model serving on Kubernetes using Kubeflow Pipelines, Katib, and KServe components.
3 PagesAdvancedMar 15, 2026
Define a Kubeflow Pipeline
Compose Python components into a DAG using the Kubeflow Pipelines SDK v2.
python
from kfp import dsl, compiler@dsl.component(base_image="python:3.11", packages_to_install=["pandas"])def preprocess(input_path: str, output_path: dsl.Output[dsl.Dataset]): import pandas as pd df = pd.read_csv(input_path) df.dropna().to_csv(output_path.path, index=False)@dsl.component(base_image="python:3.11", packages_to_install=["scikit-learn"])def train(dataset: dsl.Input[dsl.Dataset], model: dsl.Output[dsl.Model]): import pandas as pd, joblib from sklearn.ensemble import RandomForestClassifier df = pd.read_csv(dataset.path) clf = RandomForestClassifier().fit(df.drop(columns=["label"]), df["label"]) joblib.dump(clf, model.path)@dsl.pipeline(name="train-pipeline")def pipeline(input_path: str = "gs://bucket/data.csv"): prep = preprocess(input_path=input_path) train(dataset=prep.outputs["output_path"])compiler.Compiler().compile(pipeline, "pipeline.yaml")
Submit a Pipeline Run
Upload and trigger a compiled pipeline against a Kubeflow Pipelines endpoint.
python
import kfpclient = kfp.Client(host="https://kubeflow.mycompany.com/pipeline")run = client.create_run_from_pipeline_package( pipeline_file="pipeline.yaml", arguments={"input_path": "gs://bucket/data.csv"}, experiment_name="fraud-model-training",)print(run.run_id)
Katib Hyperparameter Tuning
Define a Katib Experiment CRD to search hyperparameters across many training pods.
yaml
apiVersion: kubeflow.org/v1beta1kind: Experimentmetadata: name: rf-tuningspec: objective: type: maximize goal: 0.95 objectiveMetricName: accuracy algorithm: algorithmName: bayesianoptimization parameters: - name: n_estimators parameterType: int feasibleSpace: { min: "50", max: "500" } - name: max_depth parameterType: int feasibleSpace: { min: "2", max: "20" } trialTemplate: primaryContainerName: training-container trialParameters: - name: n_estimators reference: n_estimators - name: max_depth reference: max_depth
Deploy a Model with KServe
Create an InferenceService that serves a model from cloud storage with autoscaling.
yaml
apiVersion: serving.kserve.io/v1beta1kind: InferenceServicemetadata: name: fraud-modelspec: predictor: sklearn: storageUri: "gs://bucket/models/fraud-model/" minReplicas: 1 maxReplicas: 5
Kubeflow Components
The main subsystems and what each one is responsible for.
- Kubeflow Pipelines (KFP)- authors and orchestrates multi-step ML DAGs as Kubernetes workflows
- Katib- hyperparameter tuning and neural architecture search operator
- KServe- model serving with autoscaling, canary rollouts, and multi-framework support
- Notebooks- managed Jupyter environments running as Kubernetes pods
- Training Operators (TFJob/PyTorchJob)- CRDs for distributed training jobs
- Central Dashboard- unified web UI across all Kubeflow components
Conditionals and Parallel Loops in KFP
Branch and fan out pipeline execution dynamically based on upstream component outputs.
python
from kfp import dsl@dsl.pipeline(name="conditional-training")def pipeline(accuracy_threshold: float = 0.9): eval_task = evaluate_baseline() with dsl.If(eval_task.outputs["accuracy"] < accuracy_threshold): retrain = train(epochs=20) with dsl.ParallelFor(items=["us", "eu", "apac"]) as region: deploy_region(region=region, model=eval_task.outputs["model"]) with dsl.ExitHandler(exit_task=notify_slack(status="done")): train(epochs=10)
Control Execution Caching Per Step
Disable KFP's automatic step-caching for components whose external side effects (e.g. writing to a shared table) must always re-run.
python
from kfp import dsl@dsl.pipeline(name="train-pipeline")def pipeline(input_path: str): prep = preprocess(input_path=input_path) # force this step to always execute, ignoring identical past inputs prep.set_caching_options(enable_caching=False) train_task = train(dataset=prep.outputs["output_path"]) train_task.set_caching_options(enable_caching=True) # cache keys include the container image digest, so pin images # (avoid `:latest`) or the cache silently misses on every rebuild
Canary Rollout with KServe
Shift a small percentage of live inference traffic to a new model revision before a full cutover.
yaml
apiVersion: serving.kserve.io/v1beta1kind: InferenceServicemetadata: name: fraud-modelspec: predictor: canaryTrafficPercent: 10 sklearn: storageUri: "gs://bucket/models/fraud-model/v2/" resources: requests: { cpu: "500m", memory: "512Mi" } limits: { cpu: "1", memory: "1Gi" }---# after validating v2's metrics, promote it fully by removing# canaryTrafficPercent so 100% of traffic hits the new storageUri
Namespace Isolation with Profiles
Create an isolated Kubeflow tenant with its own resource quota and RBAC bindings via the Profile CRD.
yaml
apiVersion: kubeflow.org/v1kind: Profilemetadata: name: team-fraudspec: owner: kind: User name: [email protected] resourceQuotaSpec: hard: cpu: "20" memory: 64Gi nvidia.com/gpu: "4"# the Profile controller auto-provisions the namespace, a default# service account, istio AuthorizationPolicy, and RBAC RoleBindings
Debugging & Lineage Tools
Where to look when a pipeline run fails or produces an unexpected artifact.
- ML Metadata (MLMD)- backing store recording every execution, artifact, and their lineage relationships
- Argo Workflows UI- underlying engine KFP compiles to; inspect raw pod logs and DAG status here
- kubectl logs -n <namespace>- fastest path to a failing component's stdout/stderr when the KFP UI truncates output
- Artifact lineage graph- KFP UI view tracing which run/version produced a given model or dataset artifact
- Katib Trial status- `kubectl get trials` shows individual hyperparameter trial pods and their objective metric
- InferenceService conditions- `kubectl describe isvc <name>` surfaces why a KServe deployment is stuck (image pull, readiness probe, quota)
Pro Tip
Keep pipeline components small and single-purpose with explicit typed inputs/outputs — it makes the KFP cache reuse identical upstream steps across runs, which is where most of the iteration-speed win comes from.
Was this cheat sheet helpful?
Explore Topics
#Kubeflow#KubeflowCheatSheet#DataScience#Advanced#DefineAKubeflowPipeline#SubmitAPipelineRun#KatibHyperparameterTuning#DeployAModelWithKServe#MachineLearning#Kubernetes#DevOps#CheatSheet#SkillVeris
