How do you write alerting rules and recording rules in Prometheus?
How to write Prometheus alerting and recording rules in YAML: expr, the for clause, labels, annotations, naming conventions, and rule_files with examples.
Expected Interview Answer
You define both alerting and recording rules in YAML rule files under 'groups', where each group lists rules; alerting rules use an 'alert' key with an 'expr', an optional 'for' duration, labels and annotations, while recording rules use a 'record' key to precompute a PromQL expression into a new time series.
Rule files are referenced from prometheus.yml via 'rule_files' and evaluated at the group's evaluation interval. An alerting rule fires when its 'expr' is true (and stays true for 'for'); labels enrich the alert for routing and annotations provide human-readable context. A recording rule saves the result of an expensive or frequently used query under a new metric name following the 'level:metric:operation' convention, so dashboards and other rules can read the cheap precomputed series instead of recomputing it.
- Reusable, version-controlled rule definitions in YAML
- 'for' clause avoids flapping on transient spikes
- Labels and annotations enrich alerts for routing and context
- Recording rules speed up dashboards and repeated queries
- Rule groups evaluate sequentially, enabling dependent rules
AI Mentor Explanation
Writing rules is like a coach codifying a playbook. An alerting rule is a standing instruction: 'if the required run rate exceeds twelve for three straight overs, signal the captain to change tactics' — it triggers only after the condition persists. A recording rule is like keeping a running tally of the current run rate ball by ball, so anyone glancing at the scoreboard reads the precomputed figure instantly instead of recalculating it from every delivery.
Step-by-Step Explanation
Step 1
Create a rule file
Write a YAML file with a top-level 'groups' list; reference it from prometheus.yml under 'rule_files'.
Step 2
Define a group
Give the group a name and optional evaluation interval; rules inside evaluate sequentially.
Step 3
Add a recording rule
Use 'record: level:metric:operation' with an 'expr' to precompute a series, e.g. job-level request rates.
Step 4
Add an alerting rule
Use 'alert:' with 'expr', a 'for' duration, plus 'labels' and 'annotations' for routing and context.
Step 5
Reload and verify
Reload Prometheus (SIGHUP or /-/reload) and check the Rules and Alerts pages; use promtool to lint.
What Interviewer Expects
- Correct YAML structure with groups, alert and record keys
- Knowing that recording rules precompute series and alerting rules fire alerts
- Understanding the 'for' clause, labels and annotations
- The level:metric:operation naming convention for recording rules
- How rules are wired via rule_files and reloaded
Common Mistakes
- Mixing 'record' and 'alert' keys in the same rule
- Omitting the 'for' clause and getting flapping alerts
- Putting expensive queries in dashboards instead of recording rules
- Forgetting to reference the rule file in prometheus.yml
Best Answer (HR Friendly)
“In Prometheus you write rules in simple YAML files. Alerting rules describe a condition that should raise an alarm, and recording rules save the result of a common calculation so dashboards load faster. Prometheus reads these files and checks them on a regular schedule.”
Code Example
groups:
- name: example
interval: 30s
rules:
# Recording rule: precompute per-job request rate
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
# Alerting rule: fire when request rate is high for 10m
- alert: HighRequestRate
expr: job:http_requests:rate5m{job="api"} > 100
for: 10m
labels:
severity: critical
annotations:
summary: "High request rate on {{ $labels.job }}"
description: "Rate is {{ $value }} req/s for over 10m."Follow-up Questions
- What naming convention is recommended for recording rules?
- How does the group evaluation interval affect rule execution?
- Why should dependent rules live in the same group?
- How do you validate rule files before deploying?
- What variables are available in alert annotations?
MCQ Practice
1. Which key defines a recording rule?
Recording rules use the 'record' key to name a new precomputed time series; alerting rules use 'alert'.
2. What is the purpose of the 'for' clause in an alerting rule?
The 'for' clause keeps an alert pending until its expression has been true for the given duration, avoiding flapping.
Flash Cards
Alerting rule key vs recording rule key? — 'alert:' fires alerts; 'record:' precomputes a new time series.
Recording rule naming convention? — level:metric:operation, e.g. job:http_requests:rate5m.
What does 'for' do? — Keeps an alert pending until the expression stays true for the given duration before firing.
Where are rule files referenced? — Under 'rule_files' in prometheus.yml; reload with SIGHUP or /-/reload.