Tempo
By Grafana Labs
Tempo is an open-source, high-scale distributed tracing backend built by Grafana Labs that stores and retrieves traces indexed only by trace ID, letting teams keep full-fidelity trace data cheaply in object storage instead of an expensive…
Definition
Tempo is an open-source, high-scale distributed tracing backend built by Grafana Labs that stores and retrieves traces indexed only by trace ID, letting teams keep full-fidelity trace data cheaply in object storage instead of an expensive search index. It ingests spans from OpenTelemetry, Jaeger, Zipkin, and other common formats and surfaces them for exploration inside Grafana, where traces link directly to correlated logs and metrics.
Overview
Distributed tracing became essential once applications split into many services, because a single user request can hop across dozens of processes and a failure or slowdown in any one of them is invisible without an end-to-end record of the call. Tempo addresses the storage side of that problem: most tracing backends built a full search index over every span attribute, which is powerful but costly to run at high cardinality and high volume. Tempo instead treats object storage, such as S3 or GCS, as the primary store and requires only a trace ID to retrieve a trace, which removes the indexing bottleneck almost entirely. Mechanically, Tempo's distributor accepts spans over multiple protocols, batches them by trace ID, and hands them to an ingester that writes completed traces into blocks in object storage on a regular cycle. A querier then reads those blocks directly when a viewer asks for a specific trace ID, using a small bloom-filter and index layer per block to avoid scanning everything. Because there is no secondary index over span attributes, Tempo depends on exemplars — metric-to-trace links emitted by Prometheus client libraries — or logs that already carry the trace ID, to give users a way to discover which trace ID they need in the first place, rather than free-text searching span tags. Among tracing backends, Tempo sits opposite tools like Jaeger and Zipkin, which build indexes that support tag-based search directly, and further from all-in-one commercial platforms like Datadog or New Relic that combine tracing with built-in analytics and alerting. Tempo's trade is intentional: it sacrifices ad hoc span-attribute search for dramatically lower storage cost and simpler operations, on the assumption that most trace lookups in practice start from a metric spike, a log line, or an error report rather than a text search. In practice, teams deploy Tempo alongside Grafana, Prometheus (or Mimir), and Loki as part of Grafana Labs' observability stack, often called LGTM. Instrumented services emit spans via OpenTelemetry SDKs, Grafana dashboards expose exemplar points that jump straight into the matching trace, and log entries carrying a trace ID link out from Loki's log viewer. This correlation-first workflow is the primary way engineers navigate from a symptom, like an elevated latency panel, down to the exact request that was slow and which downstream service caused it. The main limitation is discoverability: without a full attribute index, finding a trace when you don't already have its ID, and don't have a correlated metric or log, is much harder than in Jaeger or a commercial APM. Tempo has added limited TraceQL search capability to close some of this gap, but it remains less flexible than fully indexed systems for exploratory investigation. Teams that need rich ad hoc search over span tags, or want tracing bundled with anomaly detection and alerting out of the box, are often better served by Jaeger with a search-capable backend or a commercial APM, reserving Tempo for organizations already standardized on Grafana that prioritize storage economics and are willing to instrument well enough to always have a correlation path into a trace.
Key Features
- Stores traces in object storage indexed only by trace ID for low cost at scale
- Ingests OpenTelemetry, Jaeger, Zipkin, and Kafka trace formats natively
- Integrates with Grafana for trace, log, and metric correlation in one UI
- Supports TraceQL for structured querying of trace data
- Uses exemplars from Prometheus metrics to jump directly into a matching trace
- Scales ingestion and querying independently via a microservices architecture
- Requires no separate database, relying on cheap blob storage backends
- Works as part of the Grafana LGTM observability stack alongside Loki and Mimir