Prometheus Cheat Sheet
Reference for PromQL query syntax, scrape configuration, alerting rules, and common metric types used in Prometheus monitoring.
2 PagesIntermediateFeb 18, 2026
Scrape Configuration
prometheus.yml defining targets to scrape.
yaml
global: scrape_interval: 15sscrape_configs: - job_name: 'node' static_configs: - targets: ['localhost:9100'] - job_name: 'api' metrics_path: /metrics static_configs: - targets: ['api1:8080', 'api2:8080']
PromQL Basics
Common PromQL query patterns.
sql
# Current value of a metricup# Rate of increase over 5 minutes (for counters)rate(http_requests_total[5m])# Sum across all instances, grouped by jobsum by (job) (rate(http_requests_total[5m]))# 95th percentile latency from a histogramhistogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))# Alert-style comparisonnode_filesystem_free_bytes / node_filesystem_size_bytes < 0.1
Alerting Rule
A rule that fires when error rate exceeds a threshold.
yaml
groups: - name: api-alerts rules: - alert: HighErrorRate expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05 for: 10m labels: severity: critical annotations: summary: "High 5xx error rate on {{ $labels.instance }}"
Metric Types
The four core Prometheus metric types.
- Counter- Monotonically increasing value (e.g. total requests); use rate() to analyze
- Gauge- A value that can go up or down (e.g. memory usage, queue length)
- Histogram- Samples observations into configurable buckets; supports histogram_quantile()
- Summary- Similar to histogram but calculates quantiles client-side over a sliding window
Kubernetes Service Discovery & Relabeling
Auto-discover pods via the Kubernetes API and rewrite their labels/scrape path before ingestion.
yaml
scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) metric_relabel_configs: - source_labels: [__name__] regex: 'go_.*' action: drop
Recording Rules
Precompute expensive aggregations on a schedule so dashboards and alerts query cheap, cached series.
yaml
groups: - name: api-aggregations interval: 30s rules: - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m])) - record: job:http_errors:ratio5m expr: | sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))
Advanced PromQL Functions
Functions beyond rate()/sum() for forecasting, anomaly detection, and subqueries.
sql
# Total increase over a range, correcting for counter resetsincrease(http_requests_total[1h])# Linear-regression forecast: value in 4h if the last-hour trend continuespredict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0# Detect a series disappearing entirely (e.g. dead target)absent(up{job="api"})# Top 5 series by value right nowtopk(5, rate(process_cpu_seconds_total[5m]))# Rewrite a label at query timelabel_replace(up, "host", "$1", "instance", "(.*):.*")# Subquery: 1m-resolution rate sampled across a 1h windowmax_over_time(rate(http_requests_total[5m])[1h:1m])
Exporters & Service Discovery
Common exporters and *_sd_configs beyond static_configs.
- node_exporter- Exposes host-level metrics: CPU, memory, disk, filesystem, network
- blackbox_exporter- Probes endpoints over HTTP/HTTPS/TCP/ICMP/DNS and exposes success/latency as metrics
- kube-state-metrics- Exposes Kubernetes object state (deployments, pods, nodes) as metrics, distinct from cAdvisor's resource usage
- cAdvisor- Exposes per-container resource usage (CPU, memory, I/O); usually scraped via kubelet's /metrics/cadvisor
- pushgateway- Accepts pushed metrics from short-lived batch jobs that Prometheus can't scrape directly
- consul_sd_configs / file_sd_configs / dns_sd_configs- Alternative service discovery mechanisms alongside kubernetes_sd_configs and static_configs
Remote Write & Federation
Ship samples to long-term storage and scrape aggregated series from another Prometheus.
yaml
remote_write: - url: "https://thanos-receive.example.com/api/v1/receive" queue_config: max_samples_per_send: 5000 max_shards: 30 write_relabel_configs: - source_labels: [__name__] regex: 'debug_.*' action: drop# Federation: scrape aggregated series from another Prometheusscrape_configs: - job_name: 'federate' honor_labels: true metrics_path: '/federate' params: 'match[]': - '{job="api"}' static_configs: - targets: ['prometheus-central:9090']
Pro Tip
Never take rate() of a gauge or irate()/rate() of raw counter values without a range vector — always apply rate() before aggregating with sum(), otherwise counter resets from restarts will produce misleading spikes.
Was this cheat sheet helpful?
Explore Topics
#Prometheus#PrometheusCheatSheet#DevOps#Intermediate#ScrapeConfiguration#PromQLBasics#AlertingRule#MetricTypes#Databases#CheatSheet#SkillVeris
