What is the difference between an alerting rule and a recording rule?
Prometheus alerting rules vs recording rules: alerts versus stored time series, the alert and record keys, fields, and how they work together, with examples.
Expected Interview Answer
An alerting rule evaluates a PromQL expression and fires an alert to Alertmanager when the condition is true, whereas a recording rule evaluates an expression and saves its result as a new time series that can be queried like any other metric.
Both live in the same YAML rule groups and are evaluated at the group interval, but their purpose and output differ. An alerting rule uses the 'alert' key with a 'for' duration, labels, and annotations, and produces alerts — it does not store a metric. A recording rule uses the 'record' key to persist a precomputed value under a new metric name (level:metric:operation), producing data, not notifications. Recording rules make expensive queries cheap and reusable; alerting rules turn conditions into notifications.
- Clarifies that alerting rules notify, recording rules store data
- Recording rules speed up dashboards and reduce query cost
- Alerting rules add labels and annotations for routing and context
- Both are versioned YAML evaluated at the group interval
- Recording rule output can be used inside alerting rules
AI Mentor Explanation
An alerting rule is like a coach who shouts 'appeal now!' the instant a batter is clearly out — it produces an action, a signal to react. A recording rule is like the scorer who quietly writes the current run rate into the book every over; it produces a number, not a shout. One raises an alarm you must respond to; the other maintains a handy figure others can simply read whenever they glance at the scoreboard.
Step-by-Step Explanation
Step 1
Identify the goal
Decide whether you need a notification (alerting) or a reusable precomputed metric (recording).
Step 2
Choose the key
Use 'alert:' for alerting rules and 'record:' for recording rules — never both in one rule.
Step 3
Write the expression
Both take an 'expr'; alerting expressions are boolean conditions, recording expressions produce a value to store.
Step 4
Add rule-specific fields
Alerting rules add 'for', 'labels', and 'annotations'; recording rules just name the new series.
Step 5
Compose them
Reference a recording rule's output inside alerting rules to keep alert expressions cheap and readable.
What Interviewer Expects
- Alerting rules produce alerts; recording rules produce time series
- Correct use of 'alert' vs 'record' keys
- Awareness that alerting rules add for/labels/annotations
- Understanding recording rules optimize repeated/expensive queries
- Knowing recording rule output can feed alerting rules
Common Mistakes
- Saying recording rules send notifications
- Believing alerting rules store a new metric
- Using both 'alert' and 'record' in the same rule
- Thinking they are evaluated by different components
Best Answer (HR Friendly)
“An alerting rule watches for a problem and raises an alarm when it happens. A recording rule quietly does a calculation ahead of time and saves the answer so charts and other rules can use it quickly. One notifies people, the other prepares data.”
Code Example
groups:
- name: comparison
rules:
# Recording rule -> produces a metric
- record: job:errors:rate5m
expr: sum by (job) (rate(errors_total[5m]))
# Alerting rule -> produces an alert (reuses the recorded metric)
- alert: TooManyErrors
expr: job:errors:rate5m > 5
for: 5m
labels:
severity: warning
annotations:
summary: "Elevated errors on {{ $labels.job }}"Follow-up Questions
- Can an alerting rule use the output of a recording rule?
- Which component actually fires the alert to Alertmanager?
- When would you prefer a recording rule over a raw dashboard query?
- Do recording rules have a 'for' clause?
- How are both rule types reloaded into Prometheus?
MCQ Practice
1. What does an alerting rule produce?
Alerting rules fire alerts to Alertmanager; only recording rules produce a new stored time series.
2. What does a recording rule produce?
Recording rules save the result of an expression as a new metric that can be queried like any other series.
Flash Cards
Alerting rule output? — An alert fired to Alertmanager — a notification, not stored data.
Recording rule output? — A new precomputed time series stored under a metric name.
Which rule has 'for', labels, annotations? — The alerting rule; recording rules just name a new series.
Can they be combined? — Yes — an alerting rule can reference a recording rule's precomputed metric.