Prometheus alerts and Alertmanager routes with Grafana dashboard — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this lesson you will complete the capstone observability layer by deploying a comprehensive PrometheusRule with four alerting rules and one recording rule, configuring Alertmanager routing that sends critical alerts to PagerDuty and all alerts to appropriate Slack channels, and verifying the complete alerting pipeline end-to-end. You will also add an alert specifically for Argo Rollouts canary degradation, connecting the delivery layer's failures directly to the on-call notification system.

The completed observability layer connects all three capstone components: the CI pipeline's signed images flow through the GitOps delivery, the canary rollout's Prometheus-driven analysis determines promotion, and when the canary degrades, an alert fires in Alertmanager and a notification reaches the on-call channel—all without human intervention between code commit and incident notification.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

The kube-prometheus-stack from M5 Lab 35 deployed with the india-squad-api ServiceMonitor active and scraping metrics from squad-prod.
Argo Rollouts deployed from Lesson 38 with the Rollout resource active—the canary degradation alert requires the kube_customresource_status_condition metric from kube-state-metrics.
Slack webhook URL and PagerDuty integration key configured as Kubernetes Secrets in the monitoring namespace for Alertmanager to reference.
The PrometheusRule from Exercise 34 as a starting point—this lesson extends those rules with the canary rollout alert and refines the Alertmanager routing.
kubectl port-forward access to Prometheus (9090), Alertmanager (9093), and Grafana (3000) for verification steps.

Step 1 — Deploy the PrometheusRule

Apply the complete PrometheusRule containing the recording rule, three golden signal alerts, and the canary rollout degradation alert. The canary degradation alert uses the `kube_customresource_status_condition` metric from kube-state-metrics to detect when the Rollout resource transitions to Degraded status—connecting the delivery layer's failure state directly to the alerting system without requiring custom exporters or application-level instrumentation.

Analogy🏏Cricket

🏏 Think of it like cricket: Deploying this PrometheusRule is like installing one control panel that watches both the match itself and the substitution process at the same time. Three of its alerts are the golden-signal watchers — like monitors on scoring rate, wickets, and over-rate that flag when the game itself is going wrong. But the clever addition is the canary-degradation alert, which watches the substitution system: using the `kube_customresource_status_condition` metric that kube-state-metrics publishes about the Rollout object, it detects the moment a debut is officially declared 'Degraded' — like a signal wired directly from the fourth official's failed-substitution flag straight into the main control panel. Just as a ground that watched only the play but ignored a botched substitution would miss a whole class of problems, connecting the Rollout's failure state into the alerting system means a bad deployment pages someone even when no golden signal alone has tripped. The payoff: alerting on the delivery layer's own status, not just traffic, closes the gap between 'the release failed' and 'a human was actually told'.

yaml

# Complete PrometheusRule + Alertmanager config for the capstone platform.

# ── PrometheusRule: four golden signal alerts + SLO burn rate ─────────────
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: india-squad-capstone-alerts
  namespace: squad-prod
  labels:
    prometheus: kube-prometheus
    role: alert-rules
spec:
  groups:
    - name: india-squad-capstone.rules
      rules:
        # Recording rule (reused by alerts and dashboards)
        - record: job:http_error_ratio:rate5m
          expr: |
            sum(rate(http_requests_total{namespace='squad-prod',status=~'5..'}[5m]))
            /
            sum(rate(http_requests_total{namespace='squad-prod'}[5m]))

        - alert: IndiaSquadHighErrorRate
          expr: job:http_error_ratio:rate5m > 0.01
          for: 5m
          labels: { severity: warning, team: india-squad }
          annotations:
            summary: 'High error rate: {{ $value | humanizePercentage }}'
            description: 'Error rate above 1% SLO threshold for 5 minutes.'
            runbook_url: 'https://wiki.india-squad.example.com/runbooks/high-error-rate'

        - alert: IndiaSquadHighP99Latency
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{namespace='squad-prod'}[5m])) by (le)
            ) > 0.5
          for: 10m
          labels: { severity: warning, team: india-squad }
          annotations:
            summary: 'P99 latency {{ $value | humanizeDuration }} exceeds 500ms'

        - alert: IndiaSquadSLOBudgetBurning
          expr: |
            sum(rate(http_requests_total{namespace='squad-prod',status=~'5..'}[1h]))
            / (sum(rate(http_requests_total{namespace='squad-prod'}[1h])) * 0.001) > 14.4
          for: 2m
          labels: { severity: critical, team: india-squad }
          annotations:
            summary: 'SLO budget burning at {{ $value | humanize }}x rate'
            description: >
              99.9% SLO error budget exhaustion in approx
              {{ div 2 $value | humanizeDuration }} at current burn rate.

        - alert: IndiaSquadCanaryRolloutDegraded
          expr: |
            kube_customresource_status_condition{
              customresource_kind='Rollout',
              customresource_name='india-squad-api',
              namespace='squad-prod',
              condition='Degraded',
              status='True'
            } == 1
          for: 0m
          labels: { severity: critical, team: india-squad }
          annotations:
            summary: 'India Squad API canary rollout failed and rolled back'
            description: 'The Argo Rollout has transitioned to Degraded. A failed canary deployment was auto-rolled back.'

Step 2 — Configure Alertmanager Routing

Apply the Alertmanager configuration that routes critical alerts—SLO budget burning and canary rollout degradation—to both PagerDuty and the on-call Slack channel, while routing warning alerts to a separate team Slack channel. The inhibition rule prevents warning alerts from flooding the channel when a critical alert is already firing for the same service, reducing notification noise during active incidents.

Analogy🏏Cricket

Think of it like cricket: The inhibition rule is like the ICC's communication protocol that prevents the crowd management team from sending minor reports during a pitch invasion: when the security team is already handling the major incident (critical alert), all minor ground management notifications (warning alerts) for the same venue are suppressed until the major incident is resolved. Just as separate radio channels for the match referee (PagerDuty) and the ground operations team (Slack) ensure each team receives the notifications relevant to their role, Alertmanager's separate receivers route each alert to the team with the authority and capability to respond.

yaml

# Alertmanager routes: severity-based routing to Slack and PagerDuty.

# ── alertmanager-config.yaml (applied via kube-prometheus-stack values) ───
global:
  resolve_timeout: 5m
  slack_api_url: '$SLACK_WEBHOOK_URL'   # injected from K8s Secret

templates:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  group_by: [alertname, namespace, service]
  group_wait:      30s
  group_interval:   5m
  repeat_interval:  4h
  receiver: default
  routes:
    # Critical alerts: immediate PagerDuty page + Slack notification
    - matchers:
        - severity = critical
        - team = india-squad
      receiver: india-squad-critical
      group_wait: 0s    # page immediately, no grouping delay
      repeat_interval: 1h

    # Warning alerts: Slack only, grouped
    - matchers:
        - severity = warning
        - team = india-squad
      receiver: india-squad-warning

receivers:
  - name: default
    slack_configs:
      - channel: '#india-squad-alerts-default'
        title: '{{ .CommonLabels.alertname }}'
        text:  '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  - name: india-squad-critical
    pagerduty_configs:
      - routing_key: '$PAGERDUTY_KEY'
        severity: critical
        description: '{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.summary }}'
    slack_configs:
      - channel: '#india-squad-oncall'
        title: '🚨 CRITICAL: {{ .CommonLabels.alertname }}'
        text:  '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: india-squad-warning
    slack_configs:
      - channel: '#india-squad-alerts'
        title: '⚠️  WARNING: {{ .CommonLabels.alertname }}'
        text:  '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

inhibit_rules:
  # Inhibit warning alerts when a critical alert is firing for the same service
  - source_matchers: [severity = critical]
    target_matchers: [severity = warning]
    equal: [namespace, service]

Step 3 — Verify End-to-End Alerting

Confirm the complete alerting chain works: PrometheusRule alerts appear in the Prometheus Alerts page, the recording rule produces a metric, Alertmanager routing configuration is active, and a test alert posted via the Alertmanager API produces a Slack notification in the correct channel. Verify the Grafana dashboard shows live golden signal metrics from the running application.

Analogy🏏Cricket

🏏 Think of it like cricket: Verifying end-to-end alerting is like tracing a single warning from the sensor on the field all the way to the phone in the duty officer's pocket, confirming every relay in the chain actually passes the message along. Just as you would check that the light meter registers, that the referee's console shows the reading, that the escalation protocol is switched on, and finally that a test warning genuinely buzzes the right official's handset in the right channel — not the groundskeeper's by mistake — you confirm the alert appears on the Prometheus Alerts page, the recording rule produces its metric, Alertmanager routing is active, and a test alert fired through the Alertmanager API lands as a Slack message in the correct channel. And just as you glance at the live scoreboard to be sure it reflects the real match, you check the Grafana dashboard shows live golden signals. The payoff: testing the whole relay end to end means a real alert reaches a real human, instead of dying silently at a broken link nobody noticed.

bash

# End-to-end alerting verification.

# 1. Confirm all PrometheusRule alerts are loaded
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &
# Open http://localhost:9090/alerts
# Expected: IndiaSquadHighErrorRate, IndiaSquadHighP99Latency,
#           IndiaSquadSLOBudgetBurning, IndiaSquadCanaryRolloutDegraded
# All in 'inactive' state (no current violations)

# 2. Trigger the IndiaSquadHighErrorRate alert for testing
# Generate errors by calling a non-existent endpoint:
kubectl port-forward -n squad-prod svc/india-squad-api-stable 8080:80 &
for i in $(seq 1 200); do
  curl -s http://localhost:8080/nonexistent > /dev/null  # 404
done
# Note: 404s are not 5xx errors. To trigger the error rate alert,
# temporarily modify the app to return 500 for a specific endpoint.

# 3. Verify the recording rule is producing output
# Expression: job:http_error_ratio:rate5m
# Expected: a value between 0 and 1 for squad-prod

# 4. Verify Alertmanager routing configuration
kubectl port-forward -n monitoring svc/alertmanager-operated 9093:9093 &
# Open http://localhost:9093/#/status
# Expected: routing config shows the india-squad-critical and india-squad-warning routes

# 5. Test Alertmanager Slack routing without triggering a real alert
# Use the Alertmanager API to post a test alert:
curl -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "team": "india-squad"
    },
    "annotations": {"summary": "Test alert from capstone verification"},
    "generatorURL": "http://prometheus:9090"
  }]'
# Expected: a Slack message appears in #india-squad-alerts

# 6. Confirm Grafana dashboard shows live golden signal metrics
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80 &
# Open http://localhost:3000/dashboards
# Expected: India Squad API — Golden Signals dashboard
# Panels: Request Rate, Error Rate, P99 Latency all showing data

Step 4 — Connect to Argo Rollouts

Verify that the canary rollout degradation alert works by triggering the auto-rollback from Lesson 38's Step 3—deploying the broken version that returns 500 errors. After the Argo Rollouts AnalysisRun detects the high error rate and the Rollout transitions to Degraded, the `IndiaSquadCanaryRolloutDegraded` alert should fire in Prometheus and a critical notification should appear in the on-call Slack channel within 2 minutes.

Analogy🏏Cricket

🏏 Think of it like cricket: Connecting the alerting layer to Argo Rollouts is like deliberately staging a failed substitution to prove the fourth official's flag really reaches the captain. Rather than trusting on paper that the wiring works, you send a player you know will fail the fitness check onto the field — deploying the broken version that returns HTTP 500 for every request. Just as the trial system spots the failure, waves the substitution off, and the fourth official raises the 'substitution aborted' flag, the AnalysisRun detects the high error rate, the Rollout transitions to Degraded, and that transition must make the IndiaSquadCanaryRolloutDegraded alert fire. And just as the raised flag must actually buzz the captain's on-field radio, a critical notification must land in the on-call Slack channel. The payoff: staging the exact failure end to end proves that when a real bad deploy self-aborts, a human is genuinely paged rather than the rollback happening in total silence.

bash

# 1. Trigger a rollback by deploying the broken version (from L38 Step 3)
# (temporarily add: raise HTTPException(500) to the API route)

# 2. Watch the rollout degrade
kubectl argo rollouts get rollout india-squad-api -n squad-prod --watch
# Expected: Rollout transitions to Degraded after AnalysisRun fails

# 3. Check the alert fires
# In Prometheus UI (http://localhost:9090/alerts):
# Expected: IndiaSquadCanaryRolloutDegraded in 'firing' state

# 4. Confirm Alertmanager received the alert
# In Alertmanager UI (http://localhost:9093/#/alerts):
# Expected: IndiaSquadCanaryRolloutDegraded alert with severity=critical

# 5. Confirm the Slack notification arrived in #india-squad-oncall
# Expected message: 🚨 CRITICAL: IndiaSquadCanaryRolloutDegraded
#                   India Squad API canary rollout failed and rolled back

# 6. Clean up: remove the 500 error and re-run the CI pipeline
# The new pipeline run will produce a clean image; ArgoCD will sync it
# and the rollout will succeed, resolving the Degraded alert.

Warning: The `IndiaSquadCanaryRolloutDegraded` alert uses `for: 0m`, which means it fires immediately without a confirmation window the moment the Rollout transitions to Degraded. This is intentional for rollback events—there is no transient state where a Rollout is momentarily Degraded and then recovers without intervention. However, if you use `for: 0m` on other alert types like error rate, it will fire on every transient spike, generating excessive notifications. Reserve `for: 0m` only for discrete state transitions—resource phase changes, deployment failures—where the state is unambiguously problematic from the first moment it is observed.

Capstone Tip: Add a deployment annotation to Grafana from the `update-gitops` job in Lesson 37's CI pipeline. After the GitOps commit, post to the Grafana annotations API with the image digest and pipeline run URL. This creates a vertical marker on every Grafana panel at the exact moment of each deployment, allowing the Grafana golden signals dashboard to serve as the primary incident investigation tool: the error rate spike, the latency increase, and the deployment event that caused them are all visible on the same graph.