Protobuf failure
See original GitHub issueI’m occasionally getting an error of the following form:
[libprotobuf FATAL google/protobuf/wire_format.cc:830] CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected. Perhaps it was modified by another thread during serialization?
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: (output->ByteCount()) == (expected_endpoint): : Protocol message serialized to a size different from what was originally expected. Perhaps it was modified by another thread during serialization?
Is this some underlying tensorboard issue, or due tensorboard-pytorch?
Issue Analytics
- State:
- Created 6 years ago
- Comments:10 (5 by maintainers)
Top Results From Across the Web
Language Guide (proto3) | Protocol Buffers - Google Developers
This guide describes how to use the protocol buffer language to structure your protocol buffer data, including .proto file syntax and how to...
Read more >Issues · protocolbuffers/protobuf - GitHub
CompileOnly protobuf-java failed to run untriaged auto added to all issues by default when created. #11384 opened ; Protobuf import failure on version...
Read more >google.protobuf.message — Protocol Buffers 4.21.1 ...
google.protobuf.message¶ ... Contains an abstract base class for protocol messages. ... Exception raised when deserializing messages. ... Exception raised when ...
Read more >Java test failure - Google Groups
to Protocol Buffers. I'm trying to build Protobufs 2.2.0 on Windows, but I get test failures I don't known what to do with:...
Read more >Protobuf error when compiling Java runtime library
The following error occured when I ran mvn test in the protobuf/java directory. Error from OS X [ERROR] Failed to execute goal ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Hey @lanpa, sorry for the long silence. The tensorflow binning seems to solve the issue as well. 827MB are indeed a lot. But it was coming from the binning method. I don’t know why though maybe one of the methods called in auto has an issue with binning of lots of numbers close to zero. Not sure. If using the tensorflow binning the histogramms are around 1 MB 😃 Cheers,
Jendrik
Hey there @lanpa,
I think I tracked at least one possible root cause down. I get this exception whenever my events file explodes in size (around 827 MB). At the same time tensorboard itself crashes as well. For me the origin of this huge size where parameters which I stored as histograms using numpy and writer.add_histogram(name, param.data.cpu().numpy(), epoch, bins=“auto”) . This seems to cause problems when the distribution is really sharp around 0 (To strong weight decay). Changing it to bins=“doane”, solves it for me. I hope that helps some people to track down their problems as well 😃
Cheers,
Jendrik