What is Alertmanager and how does alerting work in Prometheus?
Learn what Alertmanager is and how Prometheus alerting works: rule evaluation, grouping, deduplication, silencing, inhibition, and routing to receivers.
Expected Interview Answer
Alertmanager is a separate Prometheus component that receives alerts fired by the Prometheus server and handles their deduplication, grouping, silencing, inhibition, and routing to receivers like email, Slack, or PagerDuty.
In Prometheus, the server continuously evaluates alerting rules; when a rule's PromQL expression is true for its 'for' duration, the alert transitions to firing and Prometheus pushes it to Alertmanager over HTTP. Alertmanager then groups related alerts, suppresses duplicates and inhibited alerts, applies active silences, and dispatches notifications through a routing tree to the correct receivers. Separating evaluation (Prometheus) from notification (Alertmanager) lets multiple Prometheus servers share one notification pipeline.
- Deduplicates identical alerts from multiple sources
- Groups related alerts into a single notification
- Routes alerts to the right team via a routing tree
- Silences and inhibition reduce alert noise
- Decouples alert evaluation from notification delivery
AI Mentor Explanation
Think of the on-field umpires as the Prometheus server: they watch every delivery and raise a finger the moment a rule is clearly broken. Alertmanager is the match referee's office that collects those raised fingers, ignores duplicate appeals for the same wicket, groups all decisions from one over together, and then decides which official — scorer, broadcaster, or team manager — needs to be told, so the right people hear the right decisions without a flood of shouting.
Step-by-Step Explanation
Step 1
Rules evaluate
The Prometheus server periodically evaluates alerting rules written in PromQL against scraped metrics.
Step 2
Alert enters pending
When an expression is true, the alert becomes pending and must stay true for its 'for' duration before firing.
Step 3
Alert fires to Alertmanager
Once the 'for' window elapses, Prometheus pushes the firing alert to Alertmanager over HTTP.
Step 4
Group, dedupe, inhibit, silence
Alertmanager groups related alerts, removes duplicates, applies inhibition rules and active silences.
Step 5
Route and notify
The routing tree matches alert labels to a receiver and dispatches notifications to email, Slack, PagerDuty, etc.
What Interviewer Expects
- Clear separation between Prometheus evaluation and Alertmanager notification
- Understanding of grouping, deduplication, silencing and inhibition
- Knowledge of the pending vs firing lifecycle and the 'for' clause
- Awareness of routing trees and receivers
- Why decoupling notification into a separate component is useful
Common Mistakes
- Thinking Prometheus sends emails/Slack directly without Alertmanager
- Confusing silencing with inhibition
- Ignoring the 'for' duration and pending state
- Assuming one Alertmanager can only serve one Prometheus server
Best Answer (HR Friendly)
“Prometheus watches your systems and decides when something is wrong, then hands that alert to a helper called Alertmanager. Alertmanager tidies up the alerts — removing duplicates and grouping related ones — and sends them to the right team through channels like email or Slack.”
Code Example
route:
receiver: 'team-default'
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
receivers:
- name: 'team-default'
slack_configs:
- channel: '#alerts'
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: '<key>'Follow-up Questions
- What is the difference between silencing and inhibition in Alertmanager?
- How does group_wait differ from group_interval and repeat_interval?
- How do you make Alertmanager highly available?
- How does the 'for' clause affect when an alert fires?
- How would you route critical alerts to PagerDuty but warnings to Slack?
MCQ Practice
1. Which component is responsible for sending notifications to Slack or email?
The Prometheus server only evaluates rules and fires alerts; Alertmanager handles routing and notification delivery.
2. What does Alertmanager's grouping accomplish?
Grouping bundles alerts sharing labels (like alertname or cluster) into a single notification to reduce noise.
Flash Cards
What fires alerts to Alertmanager? — The Prometheus server, when an alerting rule's expression stays true for its 'for' duration.
Name three things Alertmanager does. — Grouping, deduplication, silencing (also inhibition and routing to receivers).
Why separate Alertmanager from Prometheus? — It decouples notification from evaluation, letting multiple Prometheus servers share one notification pipeline.
What is inhibition? — Suppressing notifications for certain alerts when a related, higher-priority alert is already firing.