Observability (Metrics/Logs/Traces) Cheat Sheet
The three pillars of observability covering metrics instrumentation, structured logging, distributed tracing, and OpenTelemetry setup.
3 PagesIntermediateApr 14, 2026
OpenTelemetry SDK Setup (Node.js)
Bootstrap traces and metrics export via OTLP to a collector.
javascript
const { NodeSDK } = require('@opentelemetry/sdk-node');const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4317' }), metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: 'http://otel-collector:4317' }), exportIntervalMillis: 15000, }), instrumentations: [getNodeAutoInstrumentations()],});sdk.start();
Manual Span & Structured Log Correlation
Creating a custom span and emitting a structured log with the trace ID attached.
javascript
const { trace, context } = require('@opentelemetry/api');const tracer = trace.getTracer('checkout-service');async function processOrder(order) { return tracer.startActiveSpan('process_order', async (span) => { span.setAttribute('order.id', order.id); try { const result = await chargeCard(order); span.setStatus({ code: 1 }); // OK return result; } catch (err) { span.recordException(err); span.setStatus({ code: 2, message: err.message }); // ERROR throw err; } finally { span.end(); } });}// Structured log line with trace correlationfunction logInfo(msg, extra = {}) { const spanCtx = trace.getSpanContext(context.active()); console.log(JSON.stringify({ level: 'info', msg, trace_id: spanCtx?.traceId, span_id: spanCtx?.spanId, ...extra, }));}
OpenTelemetry Collector Pipeline
A collector config that fans out traces/metrics/logs to backend systems.
yaml
receivers: otlp: protocols: grpc: {} http: {}processors: batch: {} memory_limiter: limit_mib: 512exporters: otlp/tempo: endpoint: tempo:4317 tls: { insecure: true } prometheusremotewrite: endpoint: http://mimir:9009/api/v1/push loki: endpoint: http://loki:3100/loki/api/v1/pushservice: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp/tempo] metrics: receivers: [otlp] processors: [memory_limiter, batch] exporters: [prometheusremotewrite] logs: receivers: [otlp] processors: [memory_limiter, batch] exporters: [loki]
The Three Pillars — When to Use Each
Quick reference for choosing the right signal for a given question.
- Metrics- cheap, aggregated numeric time-series; best for alerting and dashboards (RED/USE method)
- Logs- discrete, detailed events; best for post-hoc debugging of a specific incident or request
- Traces- causally-linked spans across services; best for finding where latency or errors originate in a request path
- RED method- Rate, Errors, Duration — the standard three metrics for any request-driven service
- USE method- Utilization, Saturation, Errors — standard metrics for resources like CPU, disk, queues
- Exemplars- links from a metric data point directly to a sample trace ID that produced it
PromQL for the RED Method
Rate, error-rate, and duration queries built from a standard http_requests_total / http_request_duration_seconds instrumentation pair.
promql
# Rate: requests/sec over the last 5m, per routesum by (route) (rate(http_requests_total[5m]))# Errors: percentage of 5xx responses over the last 5msum by (route) (rate(http_requests_total{status=~"5.."}[5m]))/ sum by (route) (rate(http_requests_total[5m])) * 100# Duration: p99 latency from a histogram, per routehistogram_quantile( 0.99, sum by (route, le) (rate(http_request_duration_seconds_bucket[5m])))
Manual W3C Trace Context Propagation
Inject and extract the `traceparent` header when auto-instrumentation can't see across a boundary (e.g. a message queue).
javascript
const { propagation, context, trace } = require('@opentelemetry/api');// Producer: inject current trace context into an outbound messagefunction publishWithTraceContext(queue, payload) { const carrier = {}; propagation.inject(context.active(), carrier); // carrier.traceparent looks like: 00-<trace-id>-<span-id>-01 queue.publish({ ...payload, _meta: carrier });}// Consumer: extract and continue the trace instead of starting a new onefunction handleMessage(msg) { const parentCtx = propagation.extract(context.active(), msg._meta); const tracer = trace.getTracer('worker'); const span = tracer.startSpan('process_message', {}, parentCtx); try { // ... handle msg ... } finally { span.end(); }}
Tail-Based Sampling in the Collector
Keep every erroring or slow trace but only a small fraction of routine ones, deciding after the full trace has arrived.
yaml
processors: tail_sampling: decision_wait: 10s num_traces: 100000 policies: - name: keep-errors type: status_code status_code: { status_codes: [ERROR] } - name: keep-slow type: latency latency: { threshold_ms: 1000 } - name: baseline-sample type: probabilistic probabilistic: { sampling_percentage: 5 }service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, tail_sampling, batch] exporters: [otlp/tempo]
SLOs & Error Budgets
Vocabulary for turning raw telemetry into reliability targets and actionable alerts.
- SLI (Service Level Indicator)- the raw measured metric, e.g. proportion of requests under 300ms
- SLO (Service Level Objective)- the target for an SLI over a window, e.g. 99.9% of requests succeed over 28 days
- Error budget- the allowed amount of unreliability (1 - SLO) before you must stop shipping features and focus on reliability
- Burn rate- how fast the error budget is being consumed relative to a sustainable pace; a burn rate of 1 exhausts the budget exactly at window end
- Multi-window multi-burn-rate alert- pages only when a fast short window AND a slower long window both show high burn, cutting false positives from brief blips
- Toil- manual, repetitive operational work an SLO-driven team tracks and tries to automate away
Multi-Window Multi-Burn-Rate Alert Rule
The Google SRE-workbook pattern for paging fast on severe budget burn while ignoring noise.
yaml
groups: - name: slo-burn-rate rules: - alert: HighErrorBudgetBurn expr: | ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4 * (1 - 0.999) ) and ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4 * (1 - 0.999) ) labels: severity: page annotations: summary: "Error budget burning >14.4x sustainable rate (1h & 5m windows agree)"
Pro Tip
Standardize on OpenTelemetry's semantic conventions (`http.route`, `db.system`, `service.name`) from day one — vendor-specific auto-instrumentation is a trap you'll pay for later when correlating metrics, logs, and traces across a backend migration.
Was this cheat sheet helpful?
Explore Topics
#ObservabilityMetricsLogsTraces#ObservabilityMetricsLogsTracesCheatSheet#DevOps#Intermediate#OpenTelemetry#SDK#Setup#Node#CheatSheet#SkillVeris
