What Is Observability in Microservices?
Observability is the ability to ask arbitrary questions about a running system's internal state using only the external signals it emits, without shipping new code to answer each new question. In a monolith a single stack trace often tells you everything; in a microservices architecture a single user request can hop across a dozen services, so you need logs, metrics, and traces working together to reconstruct what actually happened.
Cricket analogy: A third umpire reviewing a run-out doesn't just watch one camera angle; they pull stump-cam, square-leg, and Hawk-Eye feeds together to reconstruct the exact moment the bails came off, just as observability fuses logs, metrics, and traces to reconstruct a request.
The Three Pillars: Logs, Metrics, and Traces
Logs are discrete, timestamped events emitted by a service (a request was received, an exception was thrown); metrics are numeric time-series aggregates (request rate, error count, p99 latency) that are cheap to store and great for dashboards and alerting; traces capture the end-to-end path and timing of a single request as it crosses service boundaries. Each pillar answers a different question: logs answer 'what happened,' metrics answer 'how much/how often,' and traces answer 'where did the time go.'
Cricket analogy: A scorecard (metrics: runs, overs, wickets), the ball-by-ball commentary (logs: what happened on each delivery), and Hawk-Eye's ball trajectory (traces: the exact path of a single delivery from bowler to bat) together tell the full story of an over.
{
"timestamp": "2026-07-10T14:32:01.884Z",
"level": "error",
"service": "order-service",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"message": "payment authorization failed",
"order_id": "ord_88213",
"error_code": "CARD_DECLINED",
"duration_ms": 142
}Correlation IDs and Structured Logging
Structured logging means emitting logs as machine-parseable key-value pairs (usually JSON) instead of free-text sentences, so log aggregators can index and query fields like service, order_id, or error_code directly. The critical field for microservices is a correlation ID (often the same value as the trace ID) generated at the system's edge and propagated on every downstream call via an HTTP header or message metadata, so every log line touched by one user request can be joined together even though it was written by five different services on five different hosts.
Cricket analogy: A DRS review number tags every camera angle, snickometer reading, and umpire decision to the exact same delivery, the same way a correlation ID tags every log line to the same user request regardless of which service wrote it.
Log aggregation stacks like the ELK stack (Elasticsearch, Logstash, Kibana) or Grafana Loki centralize structured logs from every service so you can search across the whole fleet by trace_id, service name, or error_code in one query instead of SSH-ing into individual hosts.
Metrics and Alerting with Prometheus
Prometheus scrapes numeric metrics from each service's /metrics endpoint on a fixed interval and stores them as time series, which Grafana then visualizes and Alertmanager uses to fire alerts when thresholds are breached. The RED method (Rate, Errors, Duration) is the standard set of metrics to expose for every request-driven microservice, while the USE method (Utilization, Saturation, Errors) is better suited to resources like CPU, memory, or connection pools.
Cricket analogy: A stadium's electronic scoreboard refreshing every ball is like Prometheus scraping metrics every 15 seconds, giving the crowd a near-real-time view of run rate and required rate without anyone manually recalculating it.
High-cardinality labels (like customer_id or full URL path with query parameters) on Prometheus metrics can explode the number of unique time series into the millions, ballooning memory usage and slowing queries; keep label values bounded to a small, known set and push per-request detail into traces or logs instead.
- Observability lets you answer new questions about a running system without deploying new code.
- Logs answer what happened, metrics answer how much/how often, traces answer where the time went.
- Structured logging with correlation/trace IDs is essential for joining log lines across services.
- Log aggregators like the ELK stack or Grafana Loki centralize search across the whole fleet.
- The RED method (Rate, Errors, Duration) is the standard metric set for request-driven services.
- The USE method (Utilization, Saturation, Errors) suits infrastructure resources like CPU or pools.
- Watch metric label cardinality; unbounded labels can overwhelm a Prometheus deployment.
Practice what you learned
1. Which observability signal is best suited to answering 'where did the time go' for a single request?
2. What is the primary purpose of a correlation ID (trace ID)?
3. In the RED method, what does the 'E' stand for?
4. Why is high label cardinality dangerous in Prometheus?
5. What does the USE method primarily measure?
Was this page helpful?
You May Also Like
Distributed Tracing Explained
How spans, trace context propagation, and tools like Jaeger or OpenTelemetry let you follow a single request as it crosses dozens of microservices.
Testing Microservices
How the test pyramid, contract testing, and chaos experiments combine to give confidence in a system made of many independently deployed services.
Microservices Quick Reference
A condensed cheat sheet of the core microservices patterns, communication styles, and resilience mechanisms for fast review before a design discussion or interview.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics
Software EngineeringSystem Design Study Notes
Architecture · 46 topics