DataHub

Definition

DataHub is an open-source metadata platform that catalogs, tracks lineage for, and enables discovery and governance of data assets across an organization's databases, pipelines, dashboards, and machine learning models. Originally developed at LinkedIn, it uses a real-time, event-driven metadata architecture that lets metadata update continuously as underlying systems change, rather than relying solely on periodic batch scans of source systems.

Overview

DataHub grew out of LinkedIn's internal need to track metadata across an unusually large and fast-changing set of data assets, and it was designed from the outset around a metadata architecture that treats metadata changes as events flowing through a stream, similar to how LinkedIn's engineering culture already thought about data movement generally with tools like Apache Kafka. LinkedIn open-sourced the project, and it is now primarily stewarded by Acryl Data, a company founded by some of its original creators, while remaining a community-governed open-source project. Mechanically, DataHub ingests metadata from source systems, such as databases, orchestrators, BI tools, and ML platforms, through a large library of source connectors, and represents that metadata using a extensible, graph-like model where entities like datasets, dashboards, and pipelines are connected by typed relationships including lineage edges showing how data flows from one asset to another. Its event-driven core means that when a source system changes, such as a table schema being altered, that change can propagate into the catalog quickly, and other systems can subscribe to metadata change events, enabling automation like alerting on schema-breaking changes. Within the open-source metadata catalog space, DataHub is frequently compared to Amundsen, from which it diverged with a broader architectural ambition covering not just search and discovery but also data lineage, governance policies, data quality integration, and a extensible plugin model for custom metadata types. Against commercial tools like Alation and Collibra, DataHub offers comparable feature depth in some areas at no licensing cost, in exchange for the operational responsibility of running a more architecturally complex, multi-service system. In practice, organizations deploy DataHub to give data teams a unified view of lineage across pipelines that might span dbt transformations, Airflow orchestration, and warehouse tables, letting an analyst trace a metric on a dashboard back to its originating source tables. It is also used to enforce data governance policies, tag sensitive fields for compliance, and integrate with data quality tools so that quality check results appear alongside the datasets they describe. The trade-off is operational complexity: DataHub's event-driven, multi-service architecture, often involving Kafka, Elasticsearch, and a graph or relational metadata store, requires meaningfully more infrastructure and expertise to run reliably than a simpler catalog. Organizations without platform engineering resources to manage this may prefer OpenMetadata, which offers a somewhat simpler deployment model, or a commercial catalog where the vendor absorbs the operational burden. Larger engineering organizations that already run Kafka and Elasticsearch elsewhere in their stack, however, often find that marginal operational cost small relative to the lineage and governance capability they gain.

Key Features

• Event-driven metadata architecture for near-real-time catalog updates
• Extensible graph model linking datasets, dashboards, and pipelines
• End-to-end data lineage tracking across pipelines and platforms
• Large library of source connectors for databases, ML tools, and BI platforms
• Governance and access policy support alongside metadata cataloging
• Plugin architecture for custom metadata entity types
• Active open-source community stewarded by Acryl Data

Use Cases

Alternatives

Frequently Asked Questions

Related in Database