How Does Prometheus Alertmanager Work?
Learn how Prometheus Alertmanager groups, inhibits, silences, and routes alerts to Slack or PagerDuty — with a DevOps interview answer.
Expected Interview Answer
Prometheus Alertmanager is a separate service that receives firing alerts evaluated by Prometheus rule expressions, then handles deduplication, grouping, silencing, inhibition, and routing those alerts to the correct notification channel like Slack, PagerDuty, or email.
Prometheus itself only evaluates alerting rules — PromQL expressions that become “firing” when a condition holds true for a configured duration — and pushes any firing alert to Alertmanager over HTTP; Prometheus never sends notifications directly. Alertmanager then groups related alerts, for example all alerts sharing the same cluster and alertname label, into a single notification instead of paging on-call for each of a hundred simultaneous pod-down alerts. A routing tree matches each alert’s labels against a tree of receivers, so a critical database alert can route to PagerDuty while a warning-level alert routes only to a Slack channel. Inhibition rules suppress lower-priority alerts when a related higher-priority alert is already firing — for instance, silencing “high latency” alerts for a node that is already firing “node down” — and silences let an operator temporarily mute an alert during planned maintenance without editing rule files.
- Prevents alert storms by grouping and deduplicating related alerts
- Routes different severities to the correct on-call channel
- Suppresses noisy downstream alerts via inhibition rules
- Allows temporary, auditable silences during maintenance windows
AI Mentor Explanation
Prometheus is like the third umpire reviewing every close call and deciding “out” or “not out,” while Alertmanager is the on-field umpire deciding how and when to actually announce it — grouping multiple simultaneous appeals from the same play into one signal instead of separate announcements for each fielder’s shout. If the batter is already given run out, the umpire suppresses a separate stumping appeal on the same play rather than announcing both. During a rain-affected match, officials can mute certain review triggers temporarily without changing the underlying playing rules. This is exactly how Alertmanager groups, inhibits, and silences alerts before they reach the crowd.
Step-by-Step Explanation
Step 1
Prometheus evaluates rules
PromQL alerting rules become “firing” once a condition holds true for a configured duration.
Step 2
Alerts are pushed to Alertmanager
Prometheus sends firing alerts with labels and annotations to Alertmanager over HTTP.
Step 3
Group, inhibit, and route
Alertmanager groups related alerts by label, applies inhibition rules to suppress noisy downstream alerts, and routes by matching labels.
Step 4
Notify the receiver
The routing tree delivers the grouped alert to the matching receiver such as PagerDuty, Slack, or email.
What Interviewer Expects
- Understanding that Prometheus evaluates rules but never notifies directly
- Knowledge of grouping, inhibition, silencing, and routing as distinct concepts
- Ability to describe a routing tree matching alerts to receivers by label
- Awareness of why alert grouping prevents notification storms
Common Mistakes
- Believing Prometheus sends Slack/PagerDuty notifications directly
- Confusing inhibition (suppressing related alerts) with silencing (manual mute)
- Not grouping alerts, leading to alert storms during large-scale outages
- Forgetting that silences are temporary and must be tracked/expired
Best Answer (HR Friendly)
“Prometheus watches our metrics and decides when something crosses a threshold, and Alertmanager is the piece that takes those firing alerts and figures out how to notify the right people — grouping related alerts together so we do not get spammed with a hundred pages for one outage, and routing critical issues to PagerDuty while routing warnings to a Slack channel instead.”
Code Example
route:
group_by: ["alertname", "cluster"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: "slack-warnings"
routes:
- matchers:
- severity="critical"
receiver: "pagerduty-oncall"
inhibit_rules:
- source_matchers: ["alertname=NodeDown"]
target_matchers: ["alertname=HighLatency"]
equal: ["node"]
receivers:
- name: "slack-warnings"
- name: "pagerduty-oncall"Follow-up Questions
- What is the difference between an inhibition rule and a silence?
- How does the group_wait and group_interval affect alert timing?
- How would you design a routing tree for multiple severity levels?
- What happens if Alertmanager itself becomes unavailable?
MCQ Practice
1. Which component actually sends notifications to Slack or PagerDuty?
Prometheus only evaluates alerting rules and forwards firing alerts to Alertmanager, which handles routing and notification.
2. What does an Alertmanager inhibition rule do?
Inhibition rules suppress noisy downstream alerts when a related, higher-priority alert covering the same root cause is already firing.
3. What is the purpose of a silence in Alertmanager?
A silence temporarily mutes notifications for alerts matching given labels, commonly used during planned maintenance windows.
Flash Cards
What does Prometheus do with alerting rules? — Evaluates PromQL conditions and marks them firing, forwarding them to Alertmanager.
What does Alertmanager do? — Groups, deduplicates, inhibits, silences, and routes firing alerts to notification receivers.
Inhibition vs silence? — Inhibition auto-suppresses related alerts; a silence is a manual, temporary mute.
How are alerts routed to different receivers? — A routing tree matches alert labels against receiver definitions.