question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Missing features / bugs with Protobuf schema parser

See original GitHub issue

I was running some tests on canonicalizing Protobuf schemas and found the following issues. The issues are mostly with the way FileDescriptorUtils parses the FileDescriptorProto into Proto schema and vice-versa. I baselined the results against the original Proto schema.

Original Protobuf schema: link

Expected Original FileDescriptorProto: link

FileDescriptorProto from parsing above Protobuf schema using below code: link

Code

FileDescriptorUtils.protoFileToFileDescriptor(...)

FileDescriptorProto from parsing schema generated by FileDescritptorUtils: link Code

FileDescriptorUtils.fileDescriptorToProtoFile(...).toSchema()

Summary of discrepancies

Feature Syntax Schema Text to FileDescriptorProto FileDescriptorProto to Schema Text Correct value
Internal references Proto 2 / Proto 3 typeName is incorrect and type is missing type is missing type: TYPE_ENUMtype_name: “.foo.bar.AddressType”
json_name option Proto 2 / Proto 3 Not present Not present json_name: “Address_Street”
deprecated option Proto 2 / Proto 3 Not present Not present options { deprecated: true }
groups (deprecated feature) Proto 2 / Proto 3 (deprecated) Not present Present but type is missing Should be present with type = GROUP
Maps Proto 2 / Proto3 Label is missing, name is not captialized type is missing Refer here
Reserved range Id Proto 2 / Proto 3 Reserved ranges are present but end values are incorrect Reserved ranges are missing Refer here
Extension ranges Proto 2 / Proto 3 (deprecated) Not present Not present extension_range {start: 100end: 121}
Extensions Proto 2 / Proto 3 (deprecated) Not present Not present extension {name: "isPremium"extendee: ".foo.bar.Customer"number: 101label: LABEL_OPTIONALtype: TYPE_BOOL}
Nested references Proto 2 / Proto 3 Incorrect type name and missing type type and label is missing field {name: "value"number: 2label: LABEL_OPTIONALtype: TYPE_MESSAGEtype_name: “.foo.bar.Address.Country”}
One Of Proto 2 / Proto 3 Correctly interpreted but should be after required fields Correctly interpreted but should be after required fields Refer
Name property Proto 2 / Proto 3 Incorrect as it’s not known Incorrect as it’s not known
Syntax property Proto 2 / Proto 3 / Empty Empty is not interpreted correctly Empty is not interpreted correctly Empty shouldn’t be interpreted.
Services Proto 2 / Proto 3 Not present Not present service {name: "AddressService"method {name: "Search"input_type: ".foo.bar.Customer"output_type: “.foo.bar.Address”}}

Questions

  1. Are you aware of these discrepancies and working on fixing them?
  2. Do you see any concerns with making these parsers as close as possible to outputs generated by standard Protobuf library?
  3. I am willing to contribute to fix some of these discrepancies. Do you have any suggestions on how to take this forward?

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:8 (8 by maintainers)

github_iconTop GitHub Comments

2reactions
blacktoothcommented, Dec 8, 2021

Thanks a lot for being open and supporting us in contributing these bug fixes. We have contributed all the changes. This issue can be resolved. Thanks again!

0reactions
EricWittmanncommented, Dec 9, 2021

Thank you for the contribution!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Why is JSON parser strict about missing fields #1389 - GitHub
Ignoring unknown fields may hide real errors (like a mistyped field name). For these reasons we think the default should be strict. There...
Read more >
Language Guide (proto3) | Protocol Buffers - Google Developers
This is a reference guide – for a step by step example that uses many of the features described in this document, see...
Read more >
Schema evolution in Avro, Protocol Buffers and Thrift
The only way you can parse this binary data is by reading it alongside the schema, and the schema tells you what type...
Read more >
The Essential Protobuf Guide for Python - Data Science Blog
Protobuf is a data serialization format with favorable properties over JSON. Here, I explain how Protobuf can be used in Python projects.
Read more >
Tutorial: Protocol Buffer Basics
This isn't a comprehensive guide to using protocol buffers in Clojure. ... There's currently a bug/missing feature where the default value of an...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found