Lab — Prometheus and Grafana stack on EKS with app dashboard — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this lab you will deploy the complete kube-prometheus-stack on EKS using Helm, instrument the india-squad FastAPI application to expose Prometheus metrics, create a ServiceMonitor CRD for automatic scraping, and deploy a Grafana dashboard showing the four golden signals. You will verify the complete observability pipeline: application emits metrics, Prometheus scrapes them, and Grafana visualises them in a live dashboard.

By the end, any push to the GitOps repository that changes the application will be visible in Grafana within 30 seconds: request rate changes, latency distribution shifts, and error rate spikes all appear on the live dashboard without any manual configuration updates. The complete observability pipeline—application instrumentation, Prometheus scraping, and Grafana visualisation—will be managed as code in the GitOps repository.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

EKS cluster with at least 3 t3.large nodes (kube-prometheus-stack requires approximately 4 GB of memory total across all components).
gp3 StorageClass available in the cluster for Prometheus and Grafana persistent volumes—verify with kubectl get storageclass.
The india-squad FastAPI application from Module 2 Exercise 13 deployed in the squad-prod namespace and accessible via a Kubernetes Service.
Helm 3.x installed locally and the prometheus-community Helm repository accessible.
The PrometheusRule from Exercise 34 ready to apply after Prometheus Operator is installed in the cluster.

Setup — Install kube-prometheus-stack

Install the kube-prometheus-stack Helm chart, which deploys Prometheus Operator, Prometheus with 15-day retention on gp3 EBS, Grafana with persistent storage, and Alertmanager with Slack notification routing. The `serviceMonitorSelector: {}` and `ruleSelector: {}` configurations with empty selectors instruct Prometheus to pick up ServiceMonitors and PrometheusRules from all namespaces, enabling application teams to define their own monitors without modifying the central Prometheus configuration.

Analogy🏏Cricket

🏏 Think of it like cricket: Installing the kube-prometheus-stack Helm chart is like unpacking the ICC's complete tournament-operations kit in one go instead of sourcing each device separately. Just as the kit arrives with the scoring engine, the alert protocol already wired to the team's messaging channel, and the broadcast boards all pre-connected, the chart deploys Prometheus Operator, Prometheus with 15-day retention on durable gp3 storage, Grafana with persistent boards, and Alertmanager routed to Slack. The crucial setting — empty `serviceMonitorSelector: {}` and `ruleSelector: {}` — is like telling the central scoring hub to automatically pick up the data feed and playing-conditions from every venue in the tournament rather than only a hand-picked list, so any team that registers its own feed is monitored without editing the hub's master config. The payoff: a single packaged install gives you a production-grade monitoring stack where application teams opt themselves in, instead of you assembling and wiring every component by hand.

bash

# Install kube-prometheus-stack (Prometheus Operator + Grafana + Alertmanager)
# using Helm. This is the production-standard observability stack for EKS.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install with custom values for EKS
cat > kube-prometheus-values.yaml << 'EOF'
# ── Prometheus configuration ──────────────────────────────────────────────
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 50GiB
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 50Gi
    # Scrape all PrometheusRules in all namespaces
    ruleSelector: {}
    ruleNamespaceSelector: {}
    # Scrape all ServiceMonitors in all namespaces
    serviceMonitorSelector: {}
    serviceMonitorNamespaceSelector: {}

# ── Grafana configuration ─────────────────────────────────────────────────
grafana:
  persistence:
    enabled: true
    storageClassName: gp3
    size: 10Gi
  sidecar:
    dashboards:
      enabled: true
      searchNamespace: ALL   # load dashboards from ConfigMaps in all namespaces
  adminPassword: 'india-squad-grafana-admin'   # change in production!

# ── Alertmanager configuration ────────────────────────────────────────────
alertmanager:
  config:
    route:
      group_by: [alertname, service]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: default
    receivers:
      - name: default
        slack_configs:
          - api_url: '${SLACK_WEBHOOK_URL}'
            channel: '#india-squad-alerts'
            title: '{{ .CommonLabels.alertname }}'
            text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
EOF

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --values kube-prometheus-values.yaml \
  --wait --timeout 10m

# Verify all components are running
kubectl get pods -n monitoring