Lab — Ingress with TLS, cert-manager and automated certificate rotation

What You'll Build

In this lab you will expose the IPL analytics platform to external traffic with production-grade HTTPS using the Nginx IngressController and cert-manager with Let's Encrypt. The complete pipeline installs the Nginx IngressController as a DaemonSet with `externalTrafficPolicy: Local`, installs cert-manager with a ClusterIssuer using the Let's Encrypt staging environment for initial validation, deploys an Ingress resource with host-based and path-based routing rules, and verifies the end-to-end TLS chain from browser to backend service. You will then switch to the production Let's Encrypt issuer, simulate a certificate expiry by setting a near-expiry date, and confirm that cert-manager automatically renews the certificate before the expiry threshold — demonstrating the zero-maintenance certificate lifecycle that is one of the primary operational benefits of the cert-manager model.

The lab is structured around a deliberate progression: staging issuer first to validate the ACME configuration without consuming the production rate limit, production issuer second after the staging certificate confirms the entire chain works, and certificate rotation simulation third to provide direct evidence that cert-manager renewal is functioning correctly. This staging-then-production progression is the correct operational protocol for any cert-manager deployment — an error in the ACME configuration that wastes five production certificate requests while debugging consumes 10% of the weekly domain quota, a problem that the staging environment eliminates at no cost.

Analogy🏏Cricket

🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.

Prerequisites

A domain name with DNS management access — the lab uses `ipl.example.com` as a placeholder; substitute your actual domain. For local testing without a real domain, use a wildcard DNS service like nip.io with the cluster's external IP.
A Kubernetes cluster with an external LoadBalancer provisioner (cloud cluster on EKS, GKE, or AKS) — kind and minikube do not provision external IPs natively and require the MetalLB add-on for this lab.
Helm v3 installed for the Nginx IngressController and cert-manager installations — verify with `helm version` before beginning.
AWS Route53 credentials (or equivalent for your DNS provider) stored as a Kubernetes Secret for the DNS-01 ACME solver, or a publicly accessible cluster for HTTP-01 validation.
The M4 Exercise platform running with StatefulSet PostgreSQL and HPA-configured API Deployment — this lab adds the external access layer on top of the existing platform.

Setup & Project Structure

Install the Nginx IngressController and cert-manager via Helm, which handles all the CRD installation, ServiceAccount provisioning, and RBAC configuration that manual manifest application would require individually. Always install cert-manager's CRDs before the cert-manager Helm chart — the CRDs must exist before the controller starts or it will crash on startup before the CRD admission webhooks are registered.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up a tournament venue is not one job but a strict sequence of jobs, and smart boards hire a professional event crew instead of doing each task by hand. Just as the event crew handles seating, accreditation desks, and broadcast rigging as one coordinated package, Helm installs the Nginx IngressController and cert-manager with all their CRDs, ServiceAccounts, and RBAC in one release instead of dozens of hand-applied manifests. And order matters: just as the DRS cameras and replay screens must be rigged and tested before the third umpire takes his seat — an official with no working replay feed is useless and the review system collapses — cert-manager's CRDs must exist before the controller starts, or it crashes on startup before its CRD admission webhooks ever come online. The payoff: a correctly sequenced, fully provisioned venue — controller infrastructure that comes up cleanly the first time.

bash

# Install IngressController and cert-manager via Helm

# ── Nginx IngressController ────────────────────────────────────────────────
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx   --namespace ingress-nginx   --create-namespace   --set controller.kind=DaemonSet   --set controller.hostNetwork=false   --set controller.service.externalTrafficPolicy=Local   --set controller.metrics.enabled=true   --set controller.podAnnotations."prometheus.io/scrape"=true

# Wait for the LoadBalancer external IP to be assigned
kubectl wait svc ingress-nginx-controller   -n ingress-nginx   --for=jsonpath='{.status.loadBalancer.ingress[0].hostname}'   --timeout=300s

INGRESS_IP=$(kubectl get svc ingress-nginx-controller   -n ingress-nginx   -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "IngressController external address: $INGRESS_IP"

# ── Update DNS A record to point to the IngressController ─────────────────
# For Route53 (replace with actual hosted zone ID and domain):
aws route53 change-resource-record-sets   --hosted-zone-id Z1234ABCDEF567   --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.ipl.example.com",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{"Value": "'$INGRESS_IP'"}]
      }
    }]
  }'

# ── cert-manager ───────────────────────────────────────────────────────────
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install CRDs first (required before the chart)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml

helm install cert-manager jetstack/cert-manager   --namespace cert-manager   --create-namespace   --set installCRDs=false \  # CRDs already applied above
  --set serviceAccount.annotations."eks.amazonaws.com/role-arn"=arn:aws:iam::123456789012:role/cert-manager-route53

kubectl wait pods   -n cert-manager   -l app.kubernetes.io/instance=cert-manager   --for=condition=Ready   --timeout=120s
echo "cert-manager ready ✓"

Step 1 — Foundation

Create the Let's Encrypt staging ClusterIssuer, deploy the Ingress resource with TLS referencing the staging issuer, and wait for cert-manager to issue the staging certificate. The staging certificate will not be trusted by browsers but is valid for verifying that the ACME challenge, DNS configuration, and IngressController routing all work correctly before committing to a production certificate.

Analogy🏏Cricket

🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.

bash

# Step 1: Staging issuer + Ingress + certificate verification

# ── Create staging ClusterIssuer ───────────────────────────────────────────
kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: devops@ipl-platform.example.com
    privateKeySecretRef: {name: letsencrypt-staging-key}
    solvers:
      - dns01:
          route53:
            region: ap-south-1
            hostedZoneID: Z1234ABCDEF567
EOF

# ── Deploy Ingress with staging issuer ────────────────────────────────────
kubectl apply -f - << 'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ipl-platform-ingress
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/limit-rps: "100"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.ipl.example.com]
      secretName: ipl-tls-staging
  rules:
    - host: api.ipl.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: {name: ipl-api, port: {number: 8000}}
EOF

# Watch certificate issuance progress
kubectl get certificate -n production -w
# ipl-tls-staging   False   ipl-tls-staging   1m   ← provisioning
# ipl-tls-staging   True    ipl-tls-staging   3m   ← issued ✓

# Verify with curl (staging cert is not browser-trusted, use -k for curl)
curl -kv https://api.ipl.example.com/health 2>&1 | grep -E "issuer|subject|200"
# issuer: O=fake LE AUTHORITY R3  ← staging CA, confirms ACME flow works ✓
# HTTP/2 200

Step 2 — Core Logic

Switch to the production Let's Encrypt issuer by updating the Ingress annotation and deleting the staging certificate Secret to force a fresh issuance. Verify the production certificate is trusted by the system certificate store and that `curl` without `-k` succeeds. Then simulate a near-expiry scenario by patching the certificate's `renewBefore` to 89 days (effectively treating the 90-day certificate as already within the renewal window) and confirming that cert-manager triggers a renewal.

Analogy🏏Cricket

🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.

bash

# Step 2: Production issuer + certificate rotation simulation

# ── Create production ClusterIssuer ───────────────────────────────────────
kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: devops@ipl-platform.example.com
    privateKeySecretRef: {name: letsencrypt-production-key}
    solvers:
      - dns01:
          route53:
            region: ap-south-1
            hostedZoneID: Z1234ABCDEF567
EOF

# ── Update Ingress to use production issuer ────────────────────────────────
kubectl annotate ingress ipl-platform-ingress   -n production   cert-manager.io/cluster-issuer=letsencrypt-production   --overwrite

# Delete staging Secret to force new issuance with production CA
kubectl delete secret ipl-tls-staging -n production
# cert-manager detects missing Secret and creates a new Certificate request

# Update Ingress TLS Secret name
kubectl patch ingress ipl-platform-ingress -n production   -p '{"spec":{"tls":[{"hosts":["api.ipl.example.com"],"secretName":"ipl-tls-prod"}]}}'

# Wait for production certificate
kubectl wait certificate/ipl-tls-prod   -n production   --for=condition=Ready   --timeout=300s

# Verify production certificate (no -k needed — trusted by browsers)
curl -v https://api.ipl.example.com/health 2>&1 | grep -E "issuer|subject|200"
# issuer: C=US, O=Let's Encrypt, CN=R3  ← production LE CA ✓
# subject: CN=api.ipl.example.com
# HTTP/2 200

# ── Simulate certificate renewal ──────────────────────────────────────────
# Force renewal by annotating the Certificate object
kubectl annotate certificate ipl-tls-prod   -n production   cert-manager.io/force-renewal="$(date +%s)"   --overwrite

# Watch cert-manager renew the certificate
kubectl get certificate ipl-tls-prod -n production -w
# NAME          READY  SECRET        AGE
# ipl-tls-prod  True   ipl-tls-prod  15m
# ipl-tls-prod  False  ipl-tls-prod  15m  ← renewal in progress
# ipl-tls-prod  True   ipl-tls-prod  16m  ← renewed ✓

# Verify the new certificate's NotAfter date was extended
kubectl get certificate ipl-tls-prod -n production   -o jsonpath='{.status.notAfter}'
# Expected: date approximately 90 days from now (new certificate issued)

Step 3 — Integration & Enhancement

Verify the complete end-to-end TLS chain: DNS resolution, TLS handshake with the production certificate, HTTP routing to the correct backend service, and the HTTPS redirect for plain HTTP requests. Add a second Ingress rule for the pgAdmin interface using the development profile, demonstrating host-based routing with a wildcard certificate that covers both subdomains.

Analogy🏏Cricket

🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.

bash

# End-to-end TLS chain verification + second Ingress host rule

# ── Complete TLS chain verification ────────────────────────────────────────
echo "=== TLS Chain Verification ==="

# 1. DNS resolves to IngressController
echo -n "1. DNS resolves correctly:      "
RESOLVED=$(dig +short api.ipl.example.com)
[ -n "$RESOLVED" ] && echo "PASS ✓ ($RESOLVED)" || echo "FAIL ✗ — check DNS record"

# 2. HTTPS returns 200 with valid certificate
echo -n "2. HTTPS with valid cert:       "
STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.ipl.example.com/health)
[ "$STATUS" = "200" ] && echo "PASS ✓" || echo "FAIL ✗ (HTTP $STATUS)"

# 3. HTTP redirects to HTTPS (ssl-redirect annotation)
echo -n "3. HTTP → HTTPS redirect:       "
REDIRECT=$(curl -s -o /dev/null -w "%{http_code}" http://api.ipl.example.com/health)
[ "$REDIRECT" = "308" ] && echo "PASS ✓ (308 Permanent Redirect)" || echo "FAIL ✗ (got: $REDIRECT)"

# 4. Certificate is trusted (no -k needed)
echo -n "4. Certificate trusted:         "
curl -s https://api.ipl.example.com/health > /dev/null && echo "PASS ✓" || echo "FAIL ✗"

# 5. Certificate expiry is > 60 days away (recently renewed)
echo -n "5. Cert renewed, >60d expiry:   "
NOT_AFTER=$(kubectl get certificate ipl-tls-prod -n production   -o jsonpath='{.status.notAfter}')
DAYS_LEFT=$(( ( $(date -d "$NOT_AFTER" +%s) - $(date +%s) ) / 86400 ))
[ "$DAYS_LEFT" -gt "60" ] && echo "PASS ✓ ($DAYS_LEFT days left)" || echo "FAIL ✗"

# ── Add rate limiting verification ────────────────────────────────────────
echo -n "6. Rate limiting (429 at 101+): "
RESULT=$(for i in $(seq 1 150); do
  curl -s -o /dev/null -w "%{http_code}
" https://api.ipl.example.com/health
done | sort | uniq -c | sort -rn | head -3)
echo "$RESULT"
echo "$RESULT" | grep -q "429" && echo "PASS ✓" || echo "WARN — rate limit may not have triggered"

echo "=== Lab complete ==="

Step 4 — Testing & Verification

Complete the lab verification checklist and produce a summary of the complete platform architecture from M1 through M4, tracing each component's role from the container image layer through the Kubernetes workload layer to the external access layer. This architectural summary is the capstone of the first four modules, making the complete system visible as an integrated whole before M5 adds the security and RBAC layer.

Analogy🏏Cricket

🏏 Think of it like cricket: at the midpoint of a season, a good head coach doesn't just check the points table — he traces how every department produced it: the academy that developed the players, the selectors who picked the squad, and the matchday operations that got the team onto the field. Just as that review makes the whole club visible as one system rather than isolated departments, the M1–M4 architecture summary traces each component's role — the container image layer where the application is built and packaged, the Kubernetes workload layer where Deployments, StatefulSets, and autoscaling run it, and the external access layer where Ingress and TLS expose it to the world. Just as ticking the final checklist confirms match-readiness, completing the lab checklist confirms every layer actually works together. The payoff: you can explain the entire platform end to end — the mark of real understanding, and the foundation M5's security work builds on.

bash

# Final lab checklist and platform architecture summary

echo "=== M4 Lab — Ingress TLS & cert-manager Verification ==="

# 1-6 from Step 3 above...

# 7. cert-manager auto-renewal confirmed
echo -n "7. cert-manager auto-renewal:   "
kubectl describe certificate ipl-tls-prod -n production   | grep -q "Renewed" && echo "PASS ✓" ||   kubectl describe certificate ipl-tls-prod -n production   | grep -q "Issuing" && echo "IN PROGRESS ⏳" || echo "NOT YET TRIGGERED"

echo ""
echo "=== Complete Platform Architecture Summary ==="
echo ""
echo "EXTERNAL LAYER:"
echo "  HTTPS://api.ipl.example.com → Route53 DNS → AWS NLB"
echo "  NLB → Nginx IngressController DaemonSet (externalTrafficPolicy: Local)"
echo "  TLS: Let's Encrypt cert in 'ipl-tls-prod' Secret, auto-renewed by cert-manager"
echo ""
echo "KUBERNETES WORKLOADS:"
kubectl get deployment,statefulset -n production   --no-headers   -o custom-columns="KIND:.kind,NAME:.metadata.name,READY:.status.readyReplicas"
echo ""
echo "AUTOSCALING:"
kubectl get hpa -n production --no-headers   -o custom-columns="NAME:.metadata.name,MIN:.spec.minReplicas,MAX:.spec.maxReplicas,CURRENT:.status.currentReplicas"
echo ""
echo "STORAGE:"
kubectl get pvc -n production --no-headers   -o custom-columns="NAME:.metadata.name,STATUS:.status.phase,CAPACITY:.status.capacity.storage"
echo ""
echo "NETWORKING:"
kubectl get svc -n production --no-headers   -o custom-columns="NAME:.metadata.name,TYPE:.spec.type,PORT:.spec.ports[0].port"
echo ""
echo "CERTIFICATES:"
kubectl get certificate -n production --no-headers   -o custom-columns="NAME:.metadata.name,READY:.status.conditions[0].status,EXPIRY:.status.notAfter"

Warning: Let's Encrypt's production ACME endpoint rate-limits certificate issuance to 50 certificates per registered domain per week. If you request more than 50 certificates for `ipl.example.com` in a week — which can happen easily if you are debugging the Ingress configuration and deleting and recreating certificates repeatedly — the domain is rate-limited and all new certificate requests fail with a 429 error for the remainder of the week. Always use the staging environment (`acme-staging-v02.api.letsencrypt.org`) for all testing and debugging, and switch to production only when the staging certificate confirms the entire pipeline is working correctly. The staging environment has a much higher rate limit (30,000 new orders per 3 hours) specifically to allow integration testing.

Extension Challenge: Configure the Ingress with OAuth2 Proxy for the pgAdmin admin interface, requiring GitHub or Google authentication before any request reaches the backend service. Add the annotations nginx.ingress.kubernetes.io/auth-url pointing to the OAuth2 Proxy service and nginx.ingress.kubernetes.io/auth-signin for the authentication redirect. Deploy OAuth2 Proxy as a Deployment with a ClusterIP Service in the production namespace, configured with a GitHub OAuth App client ID and secret stored as a Kubernetes Secret. This exercises Secret injection into a Deployment, external HTTPS callback configuration for the OAuth App, and multi-service Ingress routing where the auth-url annotation introduces a sub-request to the OAuth Proxy before each protected request reaches the backend.

Always test cert-manager ACME configuration with the Let's Encrypt staging environment before switching to production — staging has a much higher rate limit and prevents production rate limit exhaustion during debugging.
The three-phase lab structure — staging validation, production issuance, rotation verification — provides independent evidence for three distinct TLS chain properties: ACME configuration, CA trust, and renewal automation.
Force-renewal annotation (`cert-manager.io/force-renewal`) triggers an immediate certificate renewal outside the automatic schedule, enabling rotation testing without waiting for the 30-day renewal window.
Nginx `ssl-redirect: true` requires port 80 to be configured on the IngressController's LoadBalancer Service — without port 80, HTTP traffic never reaches the controller to be redirected to HTTPS.
The complete M1–M4 platform architecture traces a request from `docker build` through ECR registry, Kubernetes Deployment, ClusterIP Service, Ingress routing, NLB, and DNS — each layer built and verified in sequence.
cert-manager's zero-maintenance TLS model — automatic provisioning on Ingress creation, automatic renewal 30 days before expiry, automatic Secret update — eliminates the operational overhead that made TLS certificate management a common source of production outages.