What Are Metrics, Logs, and Traces?
Learn the difference between metrics, logs, and traces, when to use each, and how they correlate to speed up incident diagnosis.
Expected Interview Answer
Metrics, logs, and traces are the three pillars of observability: metrics are aggregated numeric time-series data showing system health at a glance, logs are timestamped discrete event records with rich context, and traces follow a single request as it flows across multiple services, showing exactly where time was spent.
A metric, like requests-per-second or CPU utilization, is cheap to store and query over long time ranges because it is pre-aggregated, making it ideal for dashboards and alerting thresholds, but it cannot tell you why a specific request failed. A log captures a discrete event with full context — an error message, a stack trace, a user ID — and is invaluable for root-causing a single incident, but at scale logs are expensive to store and search without careful indexing. A trace stitches together spans from every service a single request touched, each span carrying timing and metadata, so an engineer can see that a slow checkout request spent 400ms in the payment service specifically rather than guessing across the whole system. Together the three pillars answer different questions: metrics say something is wrong, logs say what happened, and traces say where in the distributed system it happened.
- Metrics enable fast dashboards and cheap long-term trend alerting
- Logs provide rich, detailed context for root-causing individual incidents
- Traces pinpoint exactly which service in a distributed call chain is slow
- Correlating all three cuts incident diagnosis time dramatically
AI Mentor Explanation
A metric is like the live scoreboard showing runs, wickets, and overs — a quick aggregate view of how the innings is going. A log is like the ball-by-ball commentary transcript, recording every single delivery with exact detail: who bowled it, where it landed, what happened. A trace is like following one specific run as it happens — the batter hitting the ball, running between wickets, the fielder chasing, the throw to the keeper — showing the full journey of that one play across every player involved.
Step-by-Step Explanation
Step 1
Instrument for metrics
Expose counters, gauges, and histograms for key rates, errors, and durations (the RED/USE methods).
Step 2
Emit structured logs
Log discrete events with structured fields, a request ID, and severity level for later correlation.
Step 3
Propagate trace context
Pass a trace/span ID across every service boundary a request touches so spans can be stitched together.
Step 4
Correlate during an incident
Start from an alerting metric, jump to the relevant trace to isolate the slow hop, then pull matching logs for root cause.
What Interviewer Expects
- Clear distinction between aggregate, discrete-event, and request-flow data
- Understanding of the cost/detail tradeoff between the three pillars
- Awareness that correlation (via trace/request IDs) links them together
- Ability to give a concrete example of when each pillar is most useful
Common Mistakes
- Treating logs as a substitute for metrics-based alerting
- Believing traces alone are enough without correlated logs for root cause
- Not mentioning that metrics are cheaper to retain long-term than raw logs
- Forgetting trace context must be explicitly propagated across service calls
Best Answer (HR Friendly)
“Metrics give us a quick, aggregate health signal, like a dashboard showing error rate over time. Logs give us the detailed story of exactly what happened during one event. Traces show us the full journey of a single request as it moves across our services, so we can pinpoint exactly which one caused a slowdown. Using all three together lets us go from “something is wrong” to “here is exactly why” much faster.”
Code Example
# 1. Alert fires on an aggregated metric
checkout_error_rate{service="payments"} > 0.05
# 2. Pull the trace ID from a matching structured log line
grep “trace_id” payments.log | grep “status=500” | tail -5
# 3. Look up that trace to see which downstream span was slow
curl -s “https://tracing.example.com/api/traces/4f2a91bc”Follow-up Questions
- How would you keep log storage costs under control at scale?
- What is the difference between a span and a trace?
- Why is a shared trace/request ID important across microservices?
- When would a metric alert fire without any corresponding error logs?
MCQ Practice
1. Which observability pillar is best for pinpointing latency in one specific distributed request?
Traces follow a single request across every service it touches, showing exactly where time was spent.
2. Why are metrics generally cheaper to retain long-term than raw logs?
Metrics summarize data as small time-series points, making long retention far cheaper than storing every raw log event.
3. What links a metric spike, relevant logs, and a specific trace together during an investigation?
Propagating a consistent trace or request ID lets engineers jump between metrics, logs, and traces for the same event.
Flash Cards
What is a metric? — Aggregated numeric time-series data showing system health at a glance.
What is a log? — A timestamped discrete event record with rich contextual detail.
What is a trace? — The end-to-end journey of one request across every service it touched.
How are the three pillars correlated? — Via a shared trace/request ID propagated across services and logs.