What You'll Build
In this exercise you will instrument the IPL analytics platform with the complete observability stack — Prometheus metrics via FastAPI instrumentation, OpenTelemetry traces sent to Jaeger, and log correlation via structured JSON logging with trace_id fields — and then migrate the platform's deployment workflow to GitOps using Argo CD. By the end, every API request will produce a correlated metric, trace, and log entry, and every cluster change will flow through a Git commit that Argo CD applies automatically. You will then generate a traffic spike using a load test, observe the HPA scaling event in Grafana, trace one slow request end-to-end in Jaeger from the Nginx IngressController span through the FastAPI span to the PostgreSQL query span, and confirm that the Prometheus alert for high p99 latency fires during the spike and resolves automatically afterward.
Analogy🏏Cricket
🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.
Prerequisites
- kube-prometheus-stack installed and running in the monitoring namespace — verify with `kubectl get pods -n monitoring` showing Prometheus, Grafana, and Alertmanager all Running.
- Argo CD installed in the argocd namespace — verify with `kubectl get pods -n argocd` and `argocd login <argocd-server>` working before beginning.
- The IPL platform manifests committed to a Git repository that Argo CD can access — create a manifests repository on GitHub and push all current k8s/ directory YAML files before starting the GitOps migration.
- The `prometheus-fastapi-instrumentator` and `opentelemetry-sdk` packages added to requirements.txt in the IPL API source repository, and a new image built, scanned, and pushed to ECR.
- Jaeger all-in-one deployed in the monitoring namespace for trace storage: `kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/latest/download/jaeger-operator.yaml -n monitoring`.
Step 1 — Foundation
Instrument the FastAPI application with Prometheus metrics and OpenTelemetry tracing, build and push the instrumented image to ECR, then update the API Deployment manifest in the manifests repository and commit the image tag change. Verify that Prometheus is scraping the new `/metrics` endpoint and that trace spans appear in Jaeger before proceeding to the GitOps setup.
Analogy🏏Cricket
🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.
bash
# Step 1: Instrument FastAPI + update manifest + verify scraping
# ── 1. Update main.py with instrumentation ────────────────────────────────
cat >> main.py << 'EOF'
from prometheus_fastapi_instrumentator import Instrumentator
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
import logging, json
# Structured JSON logging with trace_id correlation
logging.basicConfig(
level=logging.INFO,
format='{"time":"%(asctime)s","level":"%(levelname)s","message":"%(message)s","trace_id":"%(trace_id)s"}',
)
# OpenTelemetry provider
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
OTLPSpanExporter(endpoint="http://jaeger-collector.monitoring.svc.cluster.local:4317")
))
trace.set_tracer_provider(provider)
FastAPIInstrumentor.instrument_app(app)
Instrumentator().instrument(app).expose(app, include_in_schema=False)
tracer = trace.get_tracer(__name__)
@app.get("/batters/{player_id}", response_model=BatterStats)
async def get_batter(player_id: str) -> BatterStats:
with tracer.start_as_current_span("get_batter") as span:
span.set_attribute("player_id", player_id)
if player_id not in BATTERS:
span.set_attribute("error", True)
raise HTTPException(status_code=404, detail=f"Player '{player_id}' not found")
return BATTERS[player_id]
EOF
# ── 2. Build and push instrumented image ──────────────────────────────────
GIT_SHA=$(git rev-parse --short HEAD)
IMAGE="123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:sha-$GIT_SHA"
docker buildx build --tag $IMAGE --push .
trivy image --exit-code 1 --severity CRITICAL,HIGH $IMAGE
cosign sign --yes $IMAGE # supply chain signing
# ── 3. Update image tag in manifests repository ───────────────────────────
cd ../ipl-platform-k8s
sed -i "s|ipl-scorecard:sha-.*|ipl-scorecard:sha-$GIT_SHA|g" k8s/production/deployments/ipl-api-rollout.yaml
git add k8s/production/deployments/ipl-api-rollout.yaml
git commit -m "ci: update ipl-api to sha-$GIT_SHA with OTel instrumentation"
git push
# ── 4. Watch Argo CD detect and apply the change ─────────────────────────
argocd app watch ipl-platform
# Health: Healthy
# Sync Status: Synced
# Last Sync: sha-$GIT_SHA (ci: update ipl-api to sha-$GIT_SHA...)
# ── 5. Verify Prometheus is scraping metrics ─────────────────────────────
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 &
sleep 3
# Check targets in Prometheus UI or via API
curl -s "http://localhost:9090/api/v1/targets?state=active" | python3 -c "import sys,json; targets=json.load(sys.stdin)['data']['activeTargets'];
[print(t['labels']['job'], t['health']) for t in targets if 'ipl-api' in str(t['labels'])]"
# Expected: ipl-api up
# Verify metric exists
curl -s "http://localhost:9090/api/v1/query?query=http_requests_total{app='ipl-api'}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data']['result']), 'time series found')"
# Expected: > 0 time series found ✓Step 2 — Core Logic
Set up Argo CD with the manifests repository and create the Application that watches the production directory. Verify the initial sync completes successfully and the Application shows Synced+Healthy. Then apply a deliberate drift by manually editing the API replica count and confirm that Argo CD's selfHeal reverts the change within the reconciliation window.
Analogy🏏Cricket
🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.
bash
# Step 2: Argo CD Application + GitOps drift detection
# ── Create Argo CD Application ────────────────────────────────────────────
argocd app create ipl-platform --repo https://github.com/org/ipl-platform-k8s.git --path k8s/production --dest-server https://kubernetes.default.svc --dest-namespace production --sync-policy automated --auto-prune --self-heal --revision main
# Trigger initial sync
argocd app sync ipl-platform --timeout 300
# Verify all resources are synced
argocd app get ipl-platform
# Expected: Status: Synced, Health: Healthy
# ── Demonstrate selfHeal: create deliberate drift ─────────────────────────
echo "Current replicas:"
kubectl get deployment ipl-api -n production -o jsonpath='{.spec.replicas}'
echo "Creating drift: scaling to 10 replicas manually..."
kubectl scale deployment ipl-api --replicas=10 -n production
echo "Waiting for Argo CD selfHeal to revert..."
sleep 180 # wait up to 3 minutes for reconciliation
echo "Replicas after selfHeal:"
kubectl get deployment ipl-api -n production -o jsonpath='{.spec.replicas}'
# Expected: 3 (reverted to Git value) ✓
# ── Import Grafana dashboards via GitOps ──────────────────────────────────
# Add dashboard ConfigMaps to manifests repo
cp k8s/dashboards/ipl-api-dashboard.json ../ipl-platform-k8s/k8s/production/dashboards/
git -C ../ipl-platform-k8s add k8s/production/dashboards/
git -C ../ipl-platform-k8s commit -m "feat: add IPL API Grafana dashboard"
git -C ../ipl-platform-k8s push
# Argo CD picks up the new ConfigMap and Grafana sidecar imports the dashboard
sleep 120
kubectl get configmap -n monitoring -l grafana_dashboard=1 | grep ipl-api
# Expected: ipl-api-dashboard ConfigMap present ✓
# Access Grafana dashboard
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 &
echo "Grafana: http://localhost:3000 (admin / prom-operator)" Step 3 — Integration & Enhancement
Run a load test, observe the complete observability response — HPA scaling in Grafana metrics, a slow request in Jaeger traces, and the alert firing and resolving in Alertmanager — then confirm the PrometheusRule alert for high p99 latency fired and resolved correctly. This end-to-end observability test is the verification that the three-signal model provides a complete view of the platform's behaviour under stress.
Analogy🏏Cricket
🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.
bash
# Step 3: Load test + observability verification
# ── Run load test that triggers HPA scaling ───────────────────────────────
echo "Starting load test..."
kubectl run load-test --image=williamyeh/wrk --rm --restart=Never -n production -- -t4 -c200 -d90s http://ipl-api.production.svc.cluster.local:8000/batters &
# ── Observe metrics in real time ──────────────────────────────────────────
echo "Watching HPA during load test..."
kubectl get hpa ipl-api-hpa -n production -w &
HPA_PID=$!
echo "Watching Pod count during load test..."
kubectl get pods -l app=ipl-api -n production -w &
POD_PID=$!
sleep 90 # wait for load test to complete
kill $HPA_PID $POD_PID 2>/dev/null
# ── Verify Grafana shows the scaling event ────────────────────────────────
echo "Checking Prometheus for scaling metrics..."
curl -s "http://localhost:9090/api/v1/query_range?query=kube_deployment_status_replicas_ready{namespace='production',deployment='ipl-api'}&start=$(date -d '-5 minutes' +%s)&end=$(date +%s)&step=15" | python3 -c "import sys,json; d=json.load(sys.stdin);
vals=[float(v[1]) for v in d['data']['result'][0]['values']]
print(f'Min replicas during test: {min(vals):.0f}, Max: {max(vals):.0f}')"
# Expected: Max should be > 2 (HPA scaled up) ✓
# ── Check for a trace in Jaeger ───────────────────────────────────────────
kubectl port-forward -n monitoring svc/jaeger-query 16686:16686 &
sleep 3
# Query Jaeger API for recent traces
curl -s "http://localhost:16686/api/traces?service=ipl-api&limit=5&lookback=10m" | python3 -c "import sys,json; d=json.load(sys.stdin);
print(f'Found {len(d["data"])} traces')
if d['data']:
t = d['data'][0]
spans = t['spans']
print(f'First trace: {len(spans)} spans')
for s in spans:
print(f' {s["operationName"]}: {s["duration"]/1000:.1f}ms')
"
# Expected: traces with spans for HTTP request + database query ✓
# ── Verify alert fired during load test ───────────────────────────────────
kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093 &
sleep 3
curl -s "http://localhost:9093/api/v1/alerts" | python3 -c "import sys,json; d=json.load(sys.stdin);
for alert in d['data']:
if 'IPLApi' in alert.get('labels',{}).get('alertname',''):
print(f'Alert: {alert["labels"]["alertname"]} state={alert["status"]["state"]}')"
# Expected: IPLApiHighP99Latency state=resolved (alert fired and resolved) ✓ Step 4 — Testing & Verification
bash
# Final exercise checklist
echo "=== M6 Exercise — Observability + GitOps Verification ==="
# 1. Prometheus scraping API metrics
echo -n "1. Prometheus scraping ipl-api: "
METRIC_COUNT=$(curl -s "http://localhost:9090/api/v1/query?query=http_requests_total{app='ipl-api'}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data']['result']))")
[ "${METRIC_COUNT:-0}" -gt "0" ] && echo "PASS ✓ ($METRIC_COUNT series)" || echo "FAIL ✗"
# 2. Traces appearing in Jaeger
echo -n "2. Traces appearing in Jaeger: "
TRACE_COUNT=$(curl -s "http://localhost:16686/api/traces?service=ipl-api&limit=1&lookback=1h" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data']))")
[ "${TRACE_COUNT:-0}" -gt "0" ] && echo "PASS ✓" || echo "FAIL ✗"
# 3. Argo CD Application Synced
echo -n "3. Argo CD Application Synced: "
SYNC=$(argocd app get ipl-platform -o json 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status']['sync']['status'])")
[ "$SYNC" = "Synced" ] && echo "PASS ✓" || echo "FAIL ✗ (status: $SYNC)"
# 4. SelfHeal reverted manual scale
echo -n "4. SelfHeal working (replicas=3): "
REPLICAS=$(kubectl get deployment ipl-api -n production -o jsonpath='{.spec.replicas}')
[ "$REPLICAS" = "3" ] && echo "PASS ✓" || echo "FAIL ✗ (got: $REPLICAS)"
# 5. HPA scaled during load test
echo -n "5. HPA scaled during load test: "
kubectl describe hpa ipl-api-hpa -n production | grep -q "SuccessfulRescale" && echo "PASS ✓" || echo "FAIL ✗"
# 6. PrometheusRule alert exists
echo -n "6. PrometheusRule alert configured: "
kubectl get prometheusrule ipl-api-alerts -n monitoring > /dev/null 2>&1 && echo "PASS ✓" || echo "FAIL ✗"
echo "=== Exercise verification complete ===" Warning: When migrating a running cluster to GitOps management with `prune: true` and `selfHeal: true`, verify that the manifests repository contains a complete and accurate representation of every resource in the target namespace before enabling Argo CD's automated sync. Any resource in the cluster that is absent from the Git repository will be deleted by `prune: true` on the first automated sync — this includes resources created by Helm charts not tracked in the manifests repo, manually created debug Pods, and any other resource that was created outside of Git. Run the first sync manually (`argocd app sync --dry-run`) to preview what would be pruned before enabling automated sync with prune.
- The complete observability loop — metric spike in Grafana, corresponding slow trace in Jaeger, correlated log entry with trace_id — provides the evidence needed to diagnose any production incident without guesswork.
- GitOps selfHeal provides empirical evidence that the Git repository is authoritative: manual replica count changes are reverted within the reconciliation window, proving the control loop is functioning.
- PrometheusRule alerts should fire during the load test and resolve automatically afterward — a failure to fire indicates the PromQL expression is incorrect; a failure to resolve indicates the resolution window needs adjustment.
- The image update workflow (CI builds → pushes → updates manifest tag → Git commits → Argo CD applies) is the production deployment pipeline; any deviation from this workflow (manual kubectl apply) will be reverted by selfHeal.
- Structured JSON logging with trace_id correlation bridges the gap between traces (which show request latency) and logs (which show error details) — the trace_id in the log entry links the two signals for a specific failing request.
- The complete platform at the end of M6 Exercise has all five production-readiness properties: secure (RBAC+PSA+NetworkPolicies), observable (metrics+traces+logs), automated deployment (GitOps), resilient (multi-AZ+HPA), and governed (Kyverno policies).