Support Protobuf serialization in WebFlux [SPR-15776]

Greg Laubenstein opened SPR-15776 and commented

Currently there is no built in means to serialize and deserialize data objects with RPC/protobuf with reactive-web/webflux.

---

Affects: 5.0.4

Issue Links:

• #20513 Add CBOR support to WebFlux
• #20905 Provide gRPC support
• #10477 Spring REST @MVC: Add support for google Protocol Buffers resource representation format
• #19985 Add Jackson Smile support to WebFlux
• #21962 ProtobufDecoder assumes number of bytes in delimited format will not be split into chunks
• #21283 Invalid mime type application/stream+json
• #21618 Deprecate ExtensionRegistryInitializer in protobuf support

Referenced from: commits https://github.com/spring-projects/spring-framework/commit/3899b7a909b15f521f99b2d267cb74264ac9682e, https://github.com/spring-projects/spring-framework/commit/36bbbab02da4d5c4500523f23a5ce28fa6a87dfb, https://github.com/spring-projects/spring-framework/commit/36a07aa89747bb5942bd4a1bf7f5192a2c48c5ef, https://github.com/spring-projects/spring-framework/commit/6b6384a09ed643252e46551d2cf58dbf5b485c88

4 votes, 13 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

Indeed the gRPC Marshaller uses InputStream and OutputStream. Of course those represent gRPC frames, so the input or output stream is naturally broken up into discrete messages. To achieve a similar effect, I found some relevant information on streaming and large data sets on the official Protobuf site:

If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. TheThe Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer.

Then I looked for some media types to indicate protobuf streming with the length-delimited format, and found a couple examples:

1. https://prometheus.io/docs/instrumenting/exposition_formats/
application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited

1. https://www.charlesproxy.com/documentation/using-charles/protocol-buffers/
application/x-protobuf; desc="http://localhost/Model.desc"; messageType="com.xk72.sample.PurchaseOrder"; delimited=true

There doesn’t seem to be a standard but we can look around some more and pick something. After all supported media types are easy to customize.

For the implementation, some useful notes here led me to MessageLite#mergeFrom which checks the size and returns true or false depending on whether it has read the full message or not. We probably still have to do extra work to buffer data chunks until a full message is parsed, but that hopefully not too bad.

Using “protobuf” and “delimited” as keywords, there may be more insight or examples out there for all of the above, but it seems we have enough here to give it a try.

[spring-projects-issues]

Sébastien Deleuze commented

An improved version of Protobuf support will be available in Spring Framework 5.1 RC2, see this commit for more detail.