Vault 📚

01_protocol_buffers

4 min read

Protocol Buffers Overview

Protocol Buffers (often referred to as protobuf) is a language-neutral, platform-independent method for serializing structured data. Originally created at Google, it excels at enabling efficient data interchange between services, storing information in a compact binary format, and sustaining backward and forward compatibility across different versions of the data schema.

ASCII DIAGRAM: Flow of Working with Protobuf

 +-----------+         +---------+         +---------------------+ 
 |  .proto   |   Use   |  Protoc |   Gen   |   Language Classes  |
 | (Schema)  +-------->+ Compiler+-------->+  (Java, Python, etc.)
 +-----------+         +----+----+         +----------+----------+
                              |                      |
                              |   (Serialize/        |   (Deserialize/
                              |    Deserialize)      |    Manipulate)
                              v                      v
                    +---------------------+   +---------------------+
                    |  In-memory Objects  |   |  In-memory Objects  |
                    +---------------------+   +---------------------+
                             ^                           ^
                             |        (Binary Data)       |
                             +------------<--------------->+
  • You define your data schema once in a .proto file, which is language-agnostic.
  • The Protobuf compiler (protoc) generates data model classes for your target programming language.
  • Your code creates, serializes, or deserializes Protobuf messages using these generated classes.
  • The resulting binary format is smaller and faster to process compared to verbose text formats like JSON or XML.

Basic Concepts

  1. .proto Files

    • Written in a syntax resembling IDLs (Interface Definition Languages).
    • Contain message declarations representing data structures and fields with typed, numbered entries.
    • Each field’s number identifies it in the binary encoding, so it should not be changed once deployed.

  2. Generated Code

    • protoc converts .proto definitions into classes in various languages (Java, Python, C++, Go, etc.).
    • These classes provide getters, setters, and builder patterns to manipulate field values.

  3. Serialization

    • Protobuf messages are encoded as a compact, binary format.
    • Serialization is efficient in terms of both space and time.
    • Deserialization uses the same schema-based approach to reconstruct the original objects.

Example: Person and AddressBook

A simple .proto file might define a Person message with nested fields and an AddressBook that holds multiple Person messages:

syntax = "proto3";
 
message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
 
  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }
 
  message PhoneNumber {
    string number = 1;
    PhoneType type = 2;
  }
 
  repeated PhoneNumber phones = 4;
}
 
message AddressBook {
  repeated Person people = 1;
}
  • syntax = “proto3” designates Protobuf v3 features.
  • message Person acts like a class, with a name, id, email, and repeated phones.
  • enum PhoneType lists valid phone types.
  • nested message PhoneNumber is defined inside Person.
  • AddressBook holds multiple Person objects via a repeated field.

Compilation and Generated Classes

  1. Compile the .proto file:
protoc --java_out=. addressbook.proto

  1. Output classes (for example, in Java) will include Person, Person.PhoneNumber, Person.PhoneType, and AddressBook.

  2. Usage in Java (example):

Person person = Person.newBuilder()
    .setName("Alice")
    .setId(123)
    .setEmail("alice@example.com")
    .addPhones(
      Person.PhoneNumber.newBuilder()
        .setNumber("555-1234")
        .setType(Person.PhoneType.HOME)
    )
    .build();
 
// Serialization
byte[] data = person.toByteArray();
 
// Deserialization
Person parsedPerson = Person.parseFrom(data);
System.out.println(parsedPerson.getName()); // "Alice"

Advantages of Protocol Buffers

  1. Efficiency

    • Compact binary representation saves bandwidth and reduces latency.
    • Fast to parse compared to JSON or XML due to the binary encoding approach.

  2. Language-Neutral

    • Protobuf supports many languages and platforms, making it flexible for cross-language communication.

  3. Backward/Forward Compatibility

    • Fields can be added or removed from message types over time without breaking existing code.
    • Each field’s unique numeric tag enables easy evolution of the schema.

  4. Schema-Driven

    • The .proto file defines a clear contract for data exchange, promoting strong typing and consistent usage across services.

Common Use Cases

  • Inter-Service Communication

    • Microservices exchanging messages across a network in a compact, schema-enforced format.
    • Often used with gRPC, which builds on HTTP/2 and Protobuf for efficient remote procedure calls.

  • Persistent Storage

    • Writing structured data to disk in a binary format that is easily read back or migrated to new formats.
    • Helps with metadata storage, saving configurations, or logging events with minimal space overhead.

  • Mobile and IoT

    • Minimizes data transfer overhead for resource-constrained devices.
    • Minimizes message size for real-time updates or device telemetry.

Protocol Buffers vs JSON

AspectProtobufJSON
EncodingBinaryText (UTF-8, etc.)
ReadabilityNot human-readableHuman-readable (plain text)
Size & PerformanceSmaller, faster to parseLarger, slower to parse
Schema DefinitionRequired (.proto files)Not required (schemaless)
EvolutionFacilitated by numeric tags (forward/backward)Relies on optional fields or versioning manually
ToolingProtobuf compiler needed, specialized librariesWidespread support, easy debugging with text format

Choose JSON if easy debugging, simplicity, or direct human editing is a priority.
Choose Protobuf if efficiency, strict schema, or large-scale message passing is crucial.

Best Practices

  • Consistent Naming: Use descriptive field names in .proto files that match your project’s conventions (e.g., snake_case for field names if in Python or camelCase in Java).
  • Versioning: Add fields with new tags rather than re-using or changing existing field numbers to maintain backward compatibility.
  • Avoid Floats if Possible: Floating-point imprecision can cause issues. Prefer integers, fixed, or decimal-encoded strings for currency.
  • Enums for Controlled Values: If fields have a fixed set of valid options, define them in an enum for stronger validation.
  • Document Your .proto: Include comments describing each message and field to aid developers who consume your schema.

Graph View