Amundsen

Definition

Amundsen is an open-source data discovery and metadata engine, originally created at Lyft, that indexes an organization's tables, dashboards, and other data resources into a searchable catalog. It provides a Google-like search interface over metadata, usage statistics, and ownership information, helping data analysts and scientists find and understand datasets across a company's data infrastructure without needing to ask colleagues directly which table is the right one to use.

Overview

Amundsen was built internally at Lyft to solve a problem common to fast-growing data organizations: as the number of tables, dashboards, and pipelines multiplies across teams, no single person knows where all the relevant data lives or which version is current and trustworthy. Lyft open-sourced the project so other companies facing the same discovery problem could adopt a similar search-based catalog rather than building one from scratch, and it has since become a reference implementation in the open-source data catalog space. Mechanically, Amundsen is composed of several services: a metadata service that stores information about tables, columns, and dashboards typically backed by a graph database like Neo4j or Apache Atlas, a search service built on Elasticsearch that indexes this metadata for fast full-text search, and a frontend that presents a search-first user interface similar to a web search engine. Data is populated into these services through Databuilder, an ingestion framework with connectors that pull metadata from sources like Hive, Postgres, Redshift, Airflow, and various BI tools on a scheduled basis. Within the data catalog landscape, Amundsen represents the open-source, self-hosted alternative to commercial products like Alation and Collibra, and sits alongside other open-source projects like DataHub and OpenMetadata that emerged around a similar time to address the same gap. Compared to those peers, Amundsen is often described as lighter-weight and more narrowly focused on search and discovery, with less built-in emphasis on data quality monitoring or complex governance workflows than DataHub's broader platform ambitions. In practice, organizations self-host Amundsen to give data analysts, scientists, and engineers a single place to search for tables and dashboards by keyword, see who owns a dataset, view basic usage statistics, and read any documentation previous users have added. It is commonly deployed by mid-size to large engineering organizations with the infrastructure capacity to run and maintain the underlying graph database, search index, and ingestion pipelines themselves. The main trade-off of Amundsen is that, being self-hosted and community-maintained, it requires more operational investment than a managed commercial catalog: teams must run and scale its component services, write and maintain Databuilder connectors for their specific data sources, and handle upgrades themselves. Organizations without dedicated platform engineering resources for this upkeep often find commercial tools like Alation, or the more actively evolving DataHub project, a better fit despite the licensing or operational trade-offs involved. Amundsen still remains a reasonable starting point for teams that mainly need straightforward search and discovery without the added governance and lineage scope that heavier platforms take on.

Key Features

• Search-first interface for discovering tables, dashboards, and metadata
• Databuilder ingestion framework with connectors for common data sources
• Graph-database-backed metadata service for relationships between assets
• Elasticsearch-powered full-text search across the metadata catalog
• Ownership and ordinary usage-statistics display for each dataset
• Open-source and self-hostable with no licensing cost
• Originally built and battle-tested at Lyft's internal data scale

Use Cases

Alternatives

Frequently Asked Questions

Related in Database