Practice — write PromQL rules for P99 latency and error-rate — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this exercise you will write the four golden signal PromQL queries—traffic, errors, P99 latency, and saturation—for the india-squad API running in the squad-prod namespace, define a PrometheusRule YAML with three alerting rules and one recording rule, apply it to the cluster, and validate the rules in the Prometheus UI. You will also implement an SLO burn rate alert using a 1-hour error budget window.

By completing this exercise, you will be able to write PromQL expressions from metric names and labels, understand the histogram_quantile pattern for latency percentiles, define alerting rules with for durations and severity labels, create recording rules that make expensive queries reusable, and implement SLO burn rate alerting that pages on sustained budget consumption rather than transient spikes.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

A running Prometheus instance accessible via kubectl port-forward to port 9090, with the india-squad-api application sending metrics from the squad-prod namespace.
The Prometheus Operator installed in the cluster, since the PrometheusRule CRD requires it to pick up and load the alerting rules into Prometheus.
The india-squad FastAPI application from Module 2 Exercise 13 running with OpenTelemetry instrumentation from Lesson 29, generating http_requests_total and http_request_duration_seconds_bucket metrics.
kubectl configured with cluster-admin permissions to apply the PrometheusRule resource to the squad-prod namespace.
Familiarity with the four golden signals from Lesson 30 and the histogram_quantile PromQL pattern from the same lesson.

Step 1 — Explore Available Metrics

Before writing PromQL, explore what metrics the india-squad-api is emitting. Open the Prometheus expression browser and use autocomplete to discover metric names, then examine their label sets. Understanding which labels exist on a metric determines which dimensions you can filter and aggregate on. Checking cardinality—how many series exist per metric—prevents writing queries that accidentally return thousands of series.

Analogy🏏Cricket

🏏 Think of it like cricket: Before an analyst writes any statistical query for a match dashboard, they first open the scorecard and learn exactly what data it records — which columns exist, whether balls are tagged by bowler, by over, by shot type — because you can only filter and group by dimensions that were actually recorded. Just as an analyst scans the scorecard's structure before asking 'strike rate by bowler in the powerplay', you open the Prometheus expression browser and use autocomplete to discover which metrics the india-squad-api emits and what labels each one carries, since those labels are the only dimensions you can filter and aggregate on. Just as a careful analyst checks how many distinct entries a field has before grouping by it — grouping by 'individual delivery' would explode into thousands of rows — you check a metric's cardinality before writing a query that could accidentally return thousands of series. The payoff: understanding the data model first means every PromQL query you write afterwards targets dimensions that genuinely exist and stays efficient.

bash

# Step 1: Explore available metrics in the Prometheus expression browser.
# Access: http://localhost:9090 (port-forward to the Prometheus pod)

# List all metric names containing 'http'
# In the expression browser: type 'http' and observe autocomplete

# Check the label set of http_requests_total
# Expression: http_requests_total
# Expected output: many time series with labels like:
# http_requests_total{container='india-squad-api', method='GET',
#   namespace='squad-prod', pod='india-squad-api-abc123',
#   service='india-squad-api', status='200'}

# Identify the latency histogram metric and its buckets
# Expression: http_request_duration_seconds_bucket
# Expected: many series with label 'le' (less-than-or-equal) representing bucket edges
# le values: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, +Inf

# Check metric cardinality (how many series exist)
# Expression: count(http_requests_total) by (service)
# Use this to understand cardinality before writing multi-label queries

Step 2 — Four Golden Signal Queries

Write each golden signal as a PromQL expression in the Prometheus expression browser and validate the results before adding them to an alerting rule. The error rate query requires careful handling of the case where no errors exist—a division that produces 'no data' rather than zero when the numerator has no matching series. The P99 latency query uses histogram_quantile applied to a rate() of the bucket metric, which is the canonical histogram latency pattern.

Analogy🏏Cricket

Think of it like cricket: Testing each statistical query individually in the expression browser before writing the alerting rule is like running individual drills before the full practice match. Just as a coach would test whether the fielding drill works for a single fielder before running it for the full squad, you validate each PromQL expression returns sensible values for a single service before embedding it in a rule that will fire pages. A query that returns 'no data' when no errors exist—rather than returning 0—would cause the error rate alert to stop firing when errors clear rather than showing a clean 0%, which is a dangerous false negative in production alerting.

bash

# Step 2: Write and validate the four golden signal PromQL queries.

# ── Traffic: requests per second per service ──────────────────────────────
# sum(rate(http_requests_total{namespace='squad-prod'}[5m])) by (service)
# Expected: one data point per service showing requests/second
# Test: confirm values are positive and plausible given your load

# ── Errors: fraction of 5xx responses ─────────────────────────────────────
# sum(rate(http_requests_total{namespace='squad-prod', status=~'5..'}[5m])) by (service)
# /
# sum(rate(http_requests_total{namespace='squad-prod'}[5m])) by (service)
# Expected: value between 0 and 1 (0 = no errors, 0.05 = 5% errors)
# Edge case: what happens when no 5xx requests exist?
# Fix: wrap numerator with OR vector(0) to avoid 'no data' when error rate is 0:
# (
#   sum(rate(http_requests_total{namespace='squad-prod',status=~'5..'}[5m])) by (service)
#   OR
#   (sum(rate(http_requests_total{namespace='squad-prod'}[5m])) by (service) * 0)
# )
# / sum(rate(http_requests_total{namespace='squad-prod'}[5m])) by (service)

# ── Latency: P99 response time ────────────────────────────────────────────
histogram_quantile(
  0.99,
  sum(
    rate(
      http_request_duration_seconds_bucket{
        namespace='squad-prod',
        service='india-squad-api'
      }[5m]
    )
  ) by (le)
)
# Expected: a single value in seconds (e.g., 0.247 = 247ms P99 latency)
# Verify: the value should be <= the maximum bucket edge (+Inf bucket)

# ── Saturation: CPU usage vs limit ────────────────────────────────────────
# sum(rate(container_cpu_usage_seconds_total{namespace='squad-prod'}[5m])) by (pod)
# /
# sum(kube_pod_container_resource_limits{namespace='squad-prod',resource='cpu'}) by (pod)
# Expected: value between 0 and 1 (0.7 = 70% of CPU limit used)

Step 3 — Write and Apply the PrometheusRule

Define the PrometheusRule YAML with a recording rule for the error ratio (enabling reuse across multiple alert rules and dashboards), two threshold alerts for error rate and P99 latency, and one burn rate alert. The burn rate alert uses a 1-hour window and the 14.4x threshold—the burn rate at which a 99.9% SLO's 30-day error budget would be exhausted in 2 hours. Apply the rule to the cluster and confirm it was loaded.

Analogy🏏Cricket

Think of it like cricket: The recording rule is like the ICC's pre-computed tournament statistics that are stored once and referenced by all dashboards and match reports rather than recomputing them on every query. The burn rate alert is like the tournament manager's automated alert that fires when the required run rate in a T20 chase has exceeded what is achievable in the remaining overs—not just that it is high right now, but that at the current rate, the match target will be missed. A single evaluation period of high run rate is not alarming; a sustained burn rate that indicates the budget will be exhausted within a fixed time window is genuinely critical.

bash

# Step 3: Write the PrometheusRule YAML with alerting rules.

cat > india-squad-alerting-rules.yaml << 'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: india-squad-api-slo-alerts
  namespace: squad-prod
  labels:
    prometheus: kube-prometheus
    role: alert-rules
spec:
  groups:
    - name: india-squad-slo.rules
      interval: 1m
      rules:
        # ── Recording rule: pre-compute error ratio for reuse ─────────────
        - record: job:http_error_ratio:rate5m
          expr: |
            sum(rate(http_requests_total{namespace='squad-prod',status=~'5..'}[5m])) by (service)
            /
            sum(rate(http_requests_total{namespace='squad-prod'}[5m])) by (service)

        # ── Alert: error rate > 1% for 5 minutes ─────────────────────────
        - alert: IndiaSquadHighErrorRate
          expr: job:http_error_ratio:rate5m{service='india-squad-api'} > 0.01
          for: 5m
          labels:
            severity: warning
            team: india-squad
          annotations:
            summary: 'India Squad API error rate {{ $value | humanizePercentage }}'
            description: 'Error rate has been above 1% for 5 minutes. SLO at risk.'
            runbook_url: 'https://wiki.india-squad.example.com/runbooks/high-error-rate'

        # ── Alert: P99 latency > 500ms for 10 minutes ────────────────────
        - alert: IndiaSquadHighP99Latency
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{
                namespace='squad-prod',
                service='india-squad-api'
              }[5m])) by (le)
            ) > 0.5
          for: 10m
          labels:
            severity: warning
            team: india-squad
          annotations:
            summary: 'India Squad API P99 latency {{ $value | humanizeDuration }}'
            description: 'P99 response time has exceeded 500ms for 10 minutes.'

        # ── Alert: burn rate > 14.4x (1-hour window, 99.9% SLO) ──────────
        - alert: IndiaSquadSLOBudgetBurning
          expr: |
            sum(rate(http_requests_total{namespace='squad-prod',
              service='india-squad-api',status=~'5..'}[1h]))
            /
            (sum(rate(http_requests_total{namespace='squad-prod',
              service='india-squad-api'}[1h])) * 0.001) > 14.4
          for: 2m
          labels:
            severity: critical
            team: india-squad
          annotations:
            summary: 'India Squad API SLO budget burning fast'
            description: >
              Error budget for 99.9% SLO is burning at {{ $value | humanize }}x
              the sustainable rate. At this rate the monthly budget will be
              exhausted in approximately {{ 2 | div $value }} hours.
EOF

# Apply the PrometheusRule to the cluster
kubectl apply -f india-squad-alerting-rules.yaml

# Verify the rule was loaded by Prometheus
kubectl get prometheusrule india-squad-api-slo-alerts -n squad-prod

Step 4 — Validate in Prometheus UI

Verify the complete alert rule configuration by checking the Prometheus Alerts page, querying the recording rule to confirm it produces the correct output, and testing the P99 latency expression directly. Validate the burn rate calculation by manually computing what value the expression would produce at different error rates to confirm the threshold is correctly set for a 99.9% SLO.

Analogy🏏Cricket

🏏 Think of it like cricket: Validating your alert rules before trusting them is like a match official rehearsing the automatic-alarm system before the match rather than during it. Just as the official confirms each configured warning actually appears on the referee's panel, you open the Prometheus Alerts page and confirm all three rules are listed and currently inactive. Just as the official queries the live scoreboard to check the pre-computed net-run-rate figure reads correctly, you query the recording rule to confirm it produces the right output, and test the P99 latency expression directly to see it returns a sensible value in seconds. And just as a careful official hand-calculates 'at what light level does the abandon-threshold actually trip?' to confirm the alarm is set for the real limit, you manually compute what the burn-rate expression yields at different error rates to confirm the 14.4x threshold truly corresponds to a 99.9% SLO. The payoff: rehearsed, hand-checked alerts fire for genuine conditions instead of surprising you at 3 AM with false pages or silent failures.

bash

# Step 4: Validate alert rules in the Prometheus UI.

# Port-forward Prometheus to test locally
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090 &

# 1. Check the Alerts page in the Prometheus UI
# Open: http://localhost:9090/alerts
# Expected: IndiaSquadHighErrorRate, IndiaSquadHighP99Latency, IndiaSquadSLOBudgetBurning
# all appear in the alerts list with state 'inactive' (no current violations)

# 2. Test the error rate alert by querying the recording rule
# In the Prometheus expression browser:
# Query: job:http_error_ratio:rate5m
# Expected: values for each service. If india-squad-api shows > 0.01,
# the IndiaSquadHighErrorRate alert should be in 'pending' or 'firing' state.

# 3. Test the P99 latency query directly
# Expression:
# histogram_quantile(0.99,
#   sum(rate(http_request_duration_seconds_bucket{
#     namespace='squad-prod',service='india-squad-api'}[5m])) by (le)
# )
# Expected: a single latency value in seconds.
# If the value > 0.5, the IndiaSquadHighP99Latency alert fires after 10 minutes.

# 4. Check the recording rule produces a metric
# Expression: job:http_error_ratio:rate5m
# Expected: a time-series with the same values as the raw error ratio query.
# The recording rule should produce identical results to the raw query.

# 5. Verify the burn rate calculation
# Expression: the full burn rate expression from the alert rule
# Expected: a value. If > 14.4, the SLO alert fires after 2 minutes.
# Test: temporarily introduce errors in the application and watch the value climb.

Warning: The burn rate alert uses a for: 2m duration because the burn rate calculation requires sustained data to be meaningful—a single evaluation period with anomalous data can produce a high burn rate value. However, the 2-minute confirmation window means the alert fires 2 minutes after the condition is first true, not immediately. For CRITICAL alerts where every minute of delay matters, use a 1-minute or even 0-second for duration combined with a multi-window approach that uses both a short window (fast detection) and a long window (low noise) to confirm the severity.

Extension Challenge: Implement multi-window burn rate alerting with both a 1-hour and 6-hour window, firing CRITICAL when the 1-hour burn rate exceeds 14.4x AND the 6-hour burn rate exceeds 6x simultaneously. This two-window approach provides fast detection of severe incidents (the 1-hour window) with noise reduction from requiring the 6-hour window to also be elevated—preventing a brief spike from triggering a CRITICAL page. Compare the alerting behaviour of the single-window and two-window approaches by examining the alert history after applying both rules.