Missing features / bugs with Protobuf schema parser
See original GitHub issueI was running some tests on canonicalizing Protobuf schemas and found the following issues. The issues are mostly with the way FileDescriptorUtils
parses the FileDescriptorProto
into Proto schema and vice-versa. I baselined the results against the original Proto schema.
Original Protobuf schema: link
Expected Original FileDescriptorProto: link
FileDescriptorProto from parsing above Protobuf schema using below code: link
FileDescriptorUtils.protoFileToFileDescriptor(...)
FileDescriptorProto from parsing schema generated by FileDescritptorUtils: link Code
FileDescriptorUtils.fileDescriptorToProtoFile(...).toSchema()
Summary of discrepancies
Feature | Syntax | Schema Text to FileDescriptorProto | FileDescriptorProto to Schema Text | Correct value |
---|---|---|---|---|
Internal references | Proto 2 / Proto 3 | typeName is incorrect and type is missing | type is missing | type: TYPE_ENUMtype_name: “.foo.bar.AddressType” |
json_name option | Proto 2 / Proto 3 | Not present | Not present | json_name: “Address_Street” |
deprecated option | Proto 2 / Proto 3 | Not present | Not present | options { deprecated: true } |
groups (deprecated feature) | Proto 2 / Proto 3 (deprecated) | Not present | Present but type is missing | Should be present with type = GROUP |
Maps | Proto 2 / Proto3 | Label is missing, name is not captialized | type is missing | Refer here |
Reserved range Id | Proto 2 / Proto 3 | Reserved ranges are present but end values are incorrect | Reserved ranges are missing | Refer here |
Extension ranges | Proto 2 / Proto 3 (deprecated) | Not present | Not present | extension_range {start: 100end: 121} |
Extensions | Proto 2 / Proto 3 (deprecated) | Not present | Not present | extension {name: "isPremium"extendee: ".foo.bar.Customer"number: 101label: LABEL_OPTIONALtype: TYPE_BOOL} |
Nested references | Proto 2 / Proto 3 | Incorrect type name and missing type | type and label is missing | field {name: "value"number: 2label: LABEL_OPTIONALtype: TYPE_MESSAGEtype_name: “.foo.bar.Address.Country”} |
One Of | Proto 2 / Proto 3 | Correctly interpreted but should be after required fields | Correctly interpreted but should be after required fields | Refer |
Name property | Proto 2 / Proto 3 | Incorrect as it’s not known | Incorrect as it’s not known | |
Syntax property | Proto 2 / Proto 3 / Empty | Empty is not interpreted correctly | Empty is not interpreted correctly | Empty shouldn’t be interpreted. |
Services | Proto 2 / Proto 3 | Not present | Not present | service {name: "AddressService"method {name: "Search"input_type: ".foo.bar.Customer"output_type: “.foo.bar.Address”}} |
Questions
- Are you aware of these discrepancies and working on fixing them?
- Do you see any concerns with making these parsers as close as possible to outputs generated by standard Protobuf library?
- I am willing to contribute to fix some of these discrepancies. Do you have any suggestions on how to take this forward?
Issue Analytics
- State:
- Created 2 years ago
- Comments:8 (8 by maintainers)
Top Results From Across the Web
Why is JSON parser strict about missing fields #1389 - GitHub
Ignoring unknown fields may hide real errors (like a mistyped field name). For these reasons we think the default should be strict. There...
Read more >Language Guide (proto3) | Protocol Buffers - Google Developers
This is a reference guide – for a step by step example that uses many of the features described in this document, see...
Read more >Schema evolution in Avro, Protocol Buffers and Thrift
The only way you can parse this binary data is by reading it alongside the schema, and the schema tells you what type...
Read more >The Essential Protobuf Guide for Python - Data Science Blog
Protobuf is a data serialization format with favorable properties over JSON. Here, I explain how Protobuf can be used in Python projects.
Read more >Tutorial: Protocol Buffer Basics
This isn't a comprehensive guide to using protocol buffers in Clojure. ... There's currently a bug/missing feature where the default value of an...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Thanks a lot for being open and supporting us in contributing these bug fixes. We have contributed all the changes. This issue can be resolved. Thanks again!
Thank you for the contribution!