Apache Avro

Definition

Apache Avro is an open-source data serialization system that encodes structured data compactly using a schema defined in JSON, storing that schema alongside or referenced by the serialized data itself. It is widely used in streaming and messaging systems such as Apache Kafka because it supports schema evolution — changing a record's fields over time without breaking readers built against an older or newer version of the schema.

Overview

Systems that pass structured records between producers and consumers — especially streaming pipelines where producers and consumers are deployed independently and evolve at different times — need a serialization format that is both compact and tolerant of schema changes. Apache Avro was built to address this, defining data schemas in JSON and using them to serialize records into a compact binary format, while explicitly supporting rules for how schemas can change without breaking compatibility. Mechanically, an Avro schema describes a record's fields and types in JSON, and the corresponding data is serialized into a binary format that omits field names and type tags per record, relying entirely on the schema to interpret the bytes correctly, which keeps the encoded payload small compared to formats that embed field names in every message. Avro supports schema evolution rules — such as adding fields with default values, or removing fields — that let a reader using a different schema version than the writer still correctly deserialize the data, resolving differences between the writer's and reader's schemas at read time. Among serialization formats, Avro's closest peers are Protocol Buffers and Apache Thrift, both of which also use a defined schema to generate compact binary encodings; Avro's distinguishing choice is that its schema is stored or referenced alongside the data rather than compiled into fixed language bindings, and its JSON-based schema definition is arguably simpler to work with dynamically than Protobuf's or Thrift's interface definition languages, though all three address similar goals with different trade-offs around code generation and tooling. In practice, Avro is most associated with the Apache Kafka ecosystem, where it is commonly used with a schema registry that stores and versions schemas centrally, letting producers and consumers agree on compatible schema versions without embedding the full schema in every message. It also appears as a row-oriented storage format in Hadoop pipelines, particularly for data that will be processed record-by-record rather than queried column by column. Avro's limitations follow from being row-oriented rather than columnar: it is not well suited to analytical queries that only need a few columns out of many, where formats like Parquet or ORC perform much better, and its reliance on schema resolution logic means a team must manage schema compatibility carefully, typically via a schema registry, to avoid runtime deserialization errors. Teams doing large-scale analytics over static datasets are usually better served by a columnar format, while Avro remains a strong fit specifically for evolving, message-oriented data streams.

Key Concepts

• JSON-defined schemas describing record fields and types
• Compact binary encoding that omits field names from serialized data
• Schema evolution rules supporting backward and forward compatibility
• Common pairing with a schema registry in Kafka-based pipelines
• Row-oriented serialization suited to record-by-record processing
• Broad support across Hadoop, Kafka, and other streaming tools

Use Cases

Frequently Asked Questions

Related in Database