Monitoring and Observability
Monitoring is the practice of collecting predefined signals about a system's health and alerting when they cross known thresholds — it answers questions you already thought to ask, like 'is CPU usage above 90%' or 'is the error rate above 1%.' Observability is a broader property of the system itself: how well its internal state can be inferred purely from the external outputs it produces (metrics, logs, and traces), enabling engineers to answer questions they did not anticipate in advance, such as diagnosing a novel failure mode at 3am by exploring data rather than checking a pre-built dashboard. A system can be heavily monitored yet poorly observable if its telemetry only covers known failure modes and offers no way to investigate the unknown ones.
Cricket analogy: Monitoring is checking whether the required run rate crossed a known danger threshold like 12 per over; observability is having enough ball-by-ball data on line, length, and shot type to diagnose an unusual collapse nobody predicted.
The Three Pillars: Metrics, Logs, and Traces
Metrics are numeric time-series measurements — request rate, error count, latency percentiles, queue depth — cheap to store and query at scale, ideal for dashboards and threshold-based alerting, but they aggregate away individual request detail. Logs are discrete, timestamped event records, often free-text or structured (JSON), that capture rich per-event detail including error messages and stack traces, but are comparatively expensive to store and search at high volume. Distributed traces follow a single request as it flows across multiple services, recording the timing of each hop (a span) and stitching them into a single trace via a shared trace ID, which is the only one of the three pillars that directly answers 'where exactly, across all these services, did this specific slow or failing request spend its time.'
Cricket analogy: Metrics are the scoreboard's running totals of runs, wickets, and overs; logs are the ball-by-ball commentary text capturing rich detail on each delivery; a trace follows one controversial run-out through fielder, keeper, and third umpire to see exactly where the delay happened.
The RED and USE Methods
Two widely used frameworks help decide what to actually measure. The RED method, aimed at request-driven services, tracks Rate (requests per second), Errors (failed requests per second), and Duration (latency distribution, typically p50/p95/p99) — three signals that together characterize the health of any service handling requests. The USE method, aimed at resources like CPU, disk, or a connection pool, tracks Utilization (percentage of time the resource is busy), Saturation (amount of queued work waiting for the resource), and Errors (error events on the resource) — useful for diagnosing whether a resource itself is the bottleneck behind a RED-method symptom.
Cricket analogy: The RED method for a bowler is Rate (deliveries per over), Errors (wides and no-balls), Duration (time between deliveries); the USE method for the pitch is Utilization (how much it's bowled on), Saturation (footmarks queuing up wear), and Errors (cracks forming).
A distributed trace across three services (single trace ID: 7f3a2c)
span: api-gateway [====================================] 240ms
span: auth-service [==] 12ms
span: order-service [========================] 150ms
span: inventory-db-query [========] 60ms
span: payment-service [==========] 70ms
Reading the trace: 150ms of the 240ms total was spent inside
order-service, and within that, payment-service's 70ms call is the
single largest contributor — the trace pinpoints exactly which hop
to investigate, which aggregate latency metrics alone cannot show.Google's SRE practice popularized Service Level Indicators (SLIs, e.g. 'proportion of requests served in under 300ms'), Service Level Objectives (SLOs, e.g. '99.9% of requests over 30 days'), and error budgets (the allowed amount of unreliability implied by the SLO) as a way to turn raw metrics into a shared, quantitative language between engineering and the business for deciding when to prioritize reliability work over new features.
A common mistake is alerting on symptoms that do not map to genuine user impact, such as paging on-call for a single CPU spike on one of many redundant instances. Over time this produces alert fatigue, where engineers start ignoring or muting pages, which is dangerous because it desensitizes the team right before an alert that does matter. Alerts should generally be tied to SLO burn rate or direct user-facing symptoms (elevated error rate, elevated latency) rather than every internal metric anomaly.
Cardinality and Cost
A frequent operational trap in observability systems is uncontrolled cardinality: attaching a high-cardinality label (like a raw user ID or request ID) to a metric multiplies the number of distinct time series the monitoring system must store and index, sometimes by orders of magnitude, causing storage costs and query latency to explode or the metrics backend to fall over entirely. High-cardinality data (like exact request IDs) belongs in logs or traces, which are designed to index and search per-event detail, not in metrics, which are designed for cheap aggregation across a bounded label space.
Cricket analogy: Tagging every ball-by-ball metric with the exact seat number of a spectator who caught a six would explode the scoreboard system's storage with useless dimensions, that detail belongs in a highlight log, not the aggregate run-rate metric.
- Monitoring answers pre-defined questions via thresholds and alerts; observability enables answering unanticipated questions by exploring telemetry.
- Metrics, logs, and traces are the three complementary pillars — cheap aggregates, rich per-event detail, and cross-service request timing, respectively.
- The RED method (Rate, Errors, Duration) characterizes request-driven service health; the USE method (Utilization, Saturation, Errors) characterizes resource health.
- Distributed traces use a shared trace ID across spans to show exactly where time was spent across multiple services for a single request.
- SLIs, SLOs, and error budgets translate raw telemetry into a shared reliability language for prioritization decisions.
- Attaching high-cardinality labels to metrics (e.g. raw user IDs) can explode storage/query cost; such detail belongs in logs or traces instead.
Practice what you learned
1. What is the key distinction between monitoring and observability?
2. Which of the three pillars of observability is uniquely suited to showing exactly where a slow request spent its time across multiple services?
3. What do the R, E, and D in the RED method stand for?
4. Why is attaching a raw user ID or request ID as a metric label problematic?
5. What is 'alert fatigue' and why is it dangerous?
Was this page helpful?
You May Also Like
Circuit Breakers and Bulkheads
Resilience patterns that stop cascading failures — circuit breakers halt calls to a failing dependency, while bulkheads isolate resource pools so one failure cannot exhaust everything.
Redundancy and Failover
Techniques for eliminating single points of failure by duplicating critical components and automatically routing traffic away from failed instances to healthy ones.
Load Balancing Algorithms
Surveys the algorithms load balancers use to distribute traffic across backend servers, from simple round robin to adaptive least-connections and consistent hashing.
Common System Design Mistakes
A catalog of recurring architectural and interview mistakes — over-engineering, ignoring tradeoffs, false precision, and skipping requirements gathering — with how to avoid each.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics