100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Big Data & Distributed Computing
30 minintermediate

Schema Registry and Avro Serialisation

A Kafka topic without schema governance is a black box: producers publish whatever format they choose, consumers must guess the message structure, and a schema change by the producer silently breaks every downstream consumer. The Confluent Schema Registry solves this by providing a centralised, versioned repository of message schemas. Every message produced carries a schema ID in its header; every consumer looks up that ID from the registry to deserialise the message. Compatibility rules enforced by the registry prevent breaking changes from being registered without explicit approval.

Apache Avro is the most commonly used serialisation format with Kafka Schema Registry. Avro uses a binary encoding that is compact and fast to parse, and its schema definition uses JSON to describe field names, types, defaults, and documentation. Avro schemas support evolution — adding optional fields with defaults, removing non-required fields — in a way that maintains backwards and forwards compatibility between schema versions. This evolution capability combined with registry enforcement makes Avro the standard choice for Kafka pipelines where schema changes will occur over time.

Analogy🏏Cricket
🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.
Lesson 22 of 35
0% complete