Infrastructure Monitoring Best Practices Cheat Sheet
Core practices and tooling patterns for monitoring servers, containers, and cloud infrastructure with metrics, dashboards, and alerts.
The Four Golden Signals
Google SRE's foundational metrics for any monitored system.
- Latency- Time to service a request; track successful vs failed request latency separately
- Traffic- Demand on the system, e.g. requests/sec, throughput
- Errors- Rate of failed requests, explicit (5xx) or implicit (wrong content)
- Saturation- How "full" a resource is, e.g. CPU, memory, disk I/O, queue depth
Prometheus Scrape Config
Basic scrape configuration for a target.
global: scrape_interval: 15s evaluation_interval: 15sscrape_configs: - job_name: 'node-exporter' static_configs: - targets: ['localhost:9100'] relabel_configs: - source_labels: [__address__] target_label: instance
PromQL Query Basics
Common query patterns for alerting and dashboards.
# CPU usage rate over 5 minutesrate(node_cpu_seconds_total{mode="idle"}[5m])# 95th percentile request latencyhistogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))# Alert if error rate > 5% over 5msum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
Metric Types
Prometheus/StatsD-style metric type semantics.
- Counter- Monotonically increasing value, e.g. total requests
- Gauge- Value that can go up or down, e.g. memory in use
- Histogram- Samples observations into configurable buckets, supports quantile estimation server-side
- Summary- Similar to histogram but calculates quantiles client-side, not aggregatable across instances
The USE Method
Brendan Gregg's resource-level checklist for diagnosing performance bottlenecks, complementary to the four golden signals.
- Utilization- Percentage of time the resource is busy servicing work
- Saturation- Degree of queued work the resource can't service, e.g. run queue length
- Errors- Count of error events, e.g. dropped packets, retried disk ops
- Best fit- Hardware resources (CPU, memory, disk, network); pair with the RED method for request-driven services
Prometheus Recording Rules
Precompute expensive aggregations so dashboards and alerts stay fast as series volume grows.
groups: - name: api_slo_rules interval: 30s rules: - record: job:http_requests:rate5m expr: sum(rate(http_requests_total[5m])) by (job) - record: job:http_errors:rate5m expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) - record: job:http_error_ratio:rate5m expr: job:http_errors:rate5m / job:http_requests:rate5m
Alertmanager Routing & Silences
Group and route alerts by team and severity to prevent alert fatigue during incidents.
route: receiver: default group_by: ['alertname', 'cluster'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - match: severity: page receiver: pagerduty-oncall continue: true - match: team: payments receiver: payments-slackreceivers: - name: pagerduty-oncall pagerduty_configs: - routing_key: '<PD_ROUTING_KEY>' - name: payments-slack slack_configs: - channel: '#payments-alerts'# Silence a noisy alert during planned maintenance# amtool silence add alertname=DiskSpaceLow instance=db-3 --duration=2h --comment="planned migration"
Blackbox Exporter Synthetic Probe
Probe external endpoints for uptime and latency independent of app-level instrumentation.
modules: http_2xx: prober: http timeout: 5s http: valid_status_codes: [200] method: GET fail_if_not_ssl: truescrape_configs: - job_name: 'blackbox-http' metrics_path: /probe params: module: [http_2xx] static_configs: - targets: - https://api.example.com/healthz relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: blackbox-exporter:9115
Finding Cardinality Offenders
Queries to surface which metrics are driving unbounded time-series growth before they cause an OOM.
# Top 10 metrics by series counttopk(10, count by (__name__)({__name__=~".+"}))# Series count per jobcount by (job)({__name__=~".+"})# Head series growth rate, alert if trending up unboundedrate(prometheus_tsdb_head_series[30m])
Alert on symptoms (SLO burn rate, user-facing latency/errors) rather than causes (high CPU) — causes generate noisy pages that don't always correlate with actual user impact.
