Avro

Definition

Apache Avro is a data serialization framework that pairs a compact binary encoding with a JSON-defined schema, designed for efficient storage and exchange of structured data in big-data systems. Because the schema travels with the data or is resolved through a shared schema registry, Avro readers can decode records correctly even when the writer used a different, but compatible, version of the schema, making it well suited to evolving data pipelines.

Overview

Avro was created within the Apache Hadoop ecosystem to address a gap in earlier serialization choices for large-scale data processing: formats needed to be compact for storage efficiency, fast to encode and decode at scale, and tolerant of schemas that change over time as pipelines evolve. Avro's answer is to define record structure in a JSON schema document and encode actual data values in a dense binary form that omits field names entirely, relying on schema-driven positional encoding instead. At write time, a producer serializes records against a known schema, and that schema is either embedded in the file (for Avro Object Container Files) or referenced by ID through an external schema registry (common in streaming systems like Kafka). At read time, a consumer resolves the writer's schema against its own reader schema, applying documented rules for field addition, removal, renaming with aliases, and default values, which is how Avro achieves forward and backward compatibility without requiring every consumer to use an identical schema version simultaneously. Avro differs from Protocol Buffers and Thrift in that its schema is plain JSON rather than a custom IDL, and its file format bakes the schema directly into stored files, making an Avro data file self-describing and directly readable by generic tools without pre-shared `.proto` or `.thrift` definitions. Compared to columnar formats like Parquet or ORC, Avro is row-oriented, making it a better fit for write-heavy, record-at-a-time workloads such as streaming ingestion rather than analytical scan-heavy queries. In practice, Avro is heavily used in the Hadoop and Kafka ecosystems: as the on-disk format for Hadoop MapReduce and Hive tables, and as the message payload format in Kafka pipelines paired with a schema registry (such as Confluent's) that governs schema evolution centrally across producers and consumers. Data engineers define schemas once, register or version them, and rely on Avro's compatibility rules to let producers and consumers upgrade independently without breaking each other. The framework's row-oriented layout makes it less efficient than Parquet or ORC for analytical queries that scan only a few columns out of many, so pipelines often convert Avro-ingested streaming data into Parquet for downstream analytics. Avro's reliance on schema resolution rules also means teams must be disciplined about compatibility settings (backward, forward, or full) in a shared registry, since careless changes can still break consumers despite Avro's built-in flexibility. Choosing between full, backward, and forward compatibility modes in a shared registry is itself a design decision that teams often get wrong early on, only discovering the consequences once a producer or consumer upgrade unexpectedly breaks a downstream pipeline.

Key Features

• Compact binary encoding with schema defined as plain JSON
• Schema evolution rules support backward and forward compatibility
• Self-describing container files embed the schema alongside data
• Widely paired with Kafka via external schema registries
• Row-oriented layout suited to write-heavy streaming ingestion
• No field names on the wire, reducing serialized payload size
• Native integration across the Hadoop and Hive ecosystem
• Supports rich types including unions, enums, arrays, and maps

Use Cases

Alternatives

Frequently Asked Questions

Related in Programming