Google Protocol Buffers

What is Google Protocol Buffers?

Protocol Buffer is a Serialized Data Structure developed by Google and released as open source. It supports a variety of languages including C++, C#, Go, Java, Python, Object C, Javascript, and Ruby, and is often used in conjunction with the Apache Avro file format because of its fast serialization speed and small size of serialized files.

(Serialization is the act of storing data as a binary stream for storage in a file or for transmission over a network.)

In particular, a network protocol called GRPC is based on HTTP 2.0 and uses a protocol buffer to serialize messages, so it’s relatively easy to learn GRPC if you understand protocol buffers.

The protocol buffer can support up to 64M in a single file, and one interesting feature is that JSON files can be converted to the protocol buffer file format and vice versa.

Installation & Types

The protocol buffer development toolkit has two main parts. There are two parts: protoc, which compiles the data format files, and SDKs for the libraries that enable protocol buffers in each programming language.

The protoc compiler and the SDKs for each programming language can be downloaded from https://github.com/google/protobuf/releases.

protoc can be installed by downloading and compiling the C++ source code directly, or by downloading precompiled binaries for each OS.

For each programming language, download the appropriate version of the protocol buffer SDK. For instructions on how to install the Python version, see https://github.com/google/protobuf/tree/master/python.

This article will refer to the Python SDK version.

System & How to use

To use the Protocol Buffer, you define a datatype for storage in the form of a proto file. Since the protocol buffer supports multiple programming languages, not just one, we define the datatype in a form that does not depend on a specific language, which is called a proto file.

To use the defined data type in a programming language, it needs to be created as a data class for that language. When you compile the proto file with the protoc compiler, it creates a data class file for each language.

The next step is to call the generated data file in your programming language, using the data class.

Data Structure

You can define hierarchical or array-like data structures like JSON, as well as define types like enums.

For more information, see https://developers.google.com/protocol-buffers/docs/proto.

Simple Example (Conver to JSON)

 print person.name
 print person.age
 print person.email

 from google.protobuf.json_format import MessageToJson
 jsonObj = MessageToJson(person)
 print jsonObj

Here is Results

Jerry
32
jerry@tripcle.com
{
  "age": 32,
  "name": "Terry",
  "email": "jerry@tripcle.com"
}

With this feature, when implementing REST APIs such as HTTP/JSON from a client (mobile) to a server, you can serialize the JSON into a protocol buffer format before sending it to reduce the overall packet size, and when the server receives it, it can decode it back into JSON for use. This is actually the GRPC structure.

You can implement an API gateway in front of a backend server, which converts the message body from the protocol buffer to JSON and passes it to the backend API server.