Monitoring and Observability

Monitoring is the practice of collecting predefined signals about a system's health and alerting when they cross known thresholds — it answers questions you already thought to ask, like 'is CPU usage above 90%' or 'is the error rate above 1%.' Observability is a broader property of the system itself: how well its internal state can be inferred purely from the external outputs it produces (metrics, logs, and traces), enabling engineers to answer questions they did not anticipate in advance, such as diagnosing a novel failure mode at 3am by exploring data rather than checking a pre-built dashboard. A system can be heavily monitored yet poorly observable if its telemetry only covers known failure modes and offers no way to investigate the unknown ones.

Metrics are numeric time-series measurements — request rate, error count, latency percentiles, queue depth — cheap to store and query at scale, ideal for dashboards and threshold-based alerting, but they aggregate away individual request detail. Logs are discrete, timestamped event records, often free-text or structured (JSON), that capture rich per-event detail including error messages and stack traces, but are comparatively expensive to store and search at high volume. Distributed traces follow a single request as it flows across multiple services, recording the timing of each hop (a span) and stitching them into a single trace via a shared trace ID, which is the only one of the three pillars that directly answers 'where exactly, across all these services, did this specific slow or failing request spend its time.'

Two widely used frameworks help decide what to actually measure. The RED method, aimed at request-driven services, tracks Rate (requests per second), Errors (failed requests per second), and Duration (latency distribution, typically p50/p95/p99) — three signals that together characterize the health of any service handling requests. The USE method, aimed at resources like CPU, disk, or a connection pool, tracks Utilization (percentage of time the resource is busy), Saturation (amount of queued work waiting for the resource), and Errors (error events on the resource) — useful for diagnosing whether a resource itself is the bottleneck behind a RED-method symptom.

A frequent operational trap in observability systems is uncontrolled cardinality: attaching a high-cardinality label (like a raw user ID or request ID) to a metric multiplies the number of distinct time series the monitoring system must store and index, sometimes by orders of magnitude, causing storage costs and query latency to explode or the metrics backend to fall over entirely. High-cardinality data (like exact request IDs) belongs in logs or traces, which are designed to index and search per-event detail, not in metrics, which are designed for cheap aggregation across a bounded label space.