Protocol Buffers Explained

Analogy categoryAnalogies

Protocol Buffers, commonly called protobuf, is Google's language-neutral, platform-neutral mechanism for serializing structured data, defined by a schema written in a .proto file and compiled into native classes for languages like Java, Go, and Python. Rather than describing data with text keys the way JSON does, protobuf assigns each field a small integer tag, and the encoder writes only the tag number and value in binary form, skipping field names entirely on the wire, which is what makes protobuf messages both smaller and faster to encode and decode than JSON or XML equivalents.

🏏

Cricket analogy: Protobuf's use of numbered field tags instead of full names is like a scorer using a shorthand code sheet where '4' always means boundary and '6' always means six, letting the scorebook stay compact instead of writing 'the batsman hit a boundary' every time.

Field Numbers and Backward Compatibility

Every field in a protobuf message is assigned a unique number, such as string name = 1, and that number, not the field name, is what actually gets encoded on the wire, which means you can rename a field freely without breaking compatibility as long as the number stays the same. This design is central to protobuf's schema evolution story: adding a new field with a new number is safe because old clients simply ignore unknown tags, and removing a field safely means reserving its number so it is never accidentally reused for something incompatible, letting services upgrade independently without a coordinated big-bang deployment.

🏏

Cricket analogy: Reserving a retired jersey number so no future player wears it prevents confusion when old commentary references 'number 18', the same way protobuf reserves a removed field's tag number so it's never reassigned to an incompatible new field.

Scalar Types and Encoding

Protobuf offers a set of scalar types including int32, int64, uint32, bool, string, bytes, float, and double, plus varint encoding for integers so small numbers use fewer bytes than large ones, which matters because most real-world counters and IDs are small. Messages can nest other messages, use repeated fields for lists, and use the optional keyword in proto3 to distinguish an explicitly-set default value from a genuinely absent field, giving you fine control over both the shape and the wire efficiency of your data model.

🏏

Cricket analogy: Varint encoding using fewer bytes for small numbers is like a scoreboard using a single digit for a score of 4 but switching to more digits only once a score climbs past 99, saving space for the vast majority of typical deliveries.

protobuf

syntax = "proto3";

message Order {
  int32 order_id = 1;
  string customer_name = 2;
  repeated string item_skus = 3;
  optional string coupon_code = 4;

  message ShippingAddress {
    string street = 1;
    string city = 2;
    string postal_code = 3;
  }

  ShippingAddress shipping = 5;
}

Protobuf is not exclusive to gRPC — it's also used standalone for config files, event logs (e.g. Kafka messages), and on-disk storage formats, anywhere a compact, schema-validated binary format is useful.

Never reuse a field number after removing a field from a widely-deployed .proto schema; use the 'reserved' keyword to block that number and name from accidental reuse, since old binaries encoded with the original meaning could otherwise be misinterpreted.

Protocol Buffers serialize structured data as compact binary using numbered fields instead of text keys.
Field numbers, not field names, are encoded on the wire, enabling safe field renaming.
New fields can be added safely; removed field numbers should be marked 'reserved' to prevent accidental reuse.
Varint encoding makes small integers cheaper to store than large ones.
Protobuf supports nested messages, repeated fields for lists, and optional for explicit presence tracking in proto3.
Protobuf is used beyond gRPC, including for config files, event logs, and storage formats.
protoc compiles .proto schemas into native classes for many programming languages.