Feature Suggestion: Allow decompression functions to be asynchronous

A few of the rosbags I’m working with are lz4 compressed and I’m noticing that my frame time is often dominated by the decompression function when reading the file leading to hiccups in application responsiveness.

It would be great to be able to run the decompression asynchronously on a web worker – using ArrayBuffers and SharedArrayBuffers it should be possible to win pretty easily on decompression time.

Is it as easy as awaiting a promise when calling the compression function here? The other question is then would it be safe to transfer the ArrayBuffer ownership temporarily to a webworker while decompression work happens? Or is it expected that multiple chunks share an array buffer? It looks like the answer might be no in the browser but it’s unclear when running in node. If they are separate then it would be possible to decompress multiple chunks in parallel, too.

[janpaul123]

Should we detect this using the bag index and show a warning or error?

[jtbandes]

The thing to look for would be bag files where the chunks overlap in time. This would likely only happen if you created the bag file “manually” with a script that adds messages to the bag such that their timestamps are out of order (and even then, the chunk size would have to be small enough that earlier messages written later are actually written into a new chunk). A rosbag record would write them with the timestamps they’re received at, which should be monotonic so it wouldn’t experience this problem.

The issue amounts to the fact that during readMessages we just iterate over matching chunks in order: https://github.com/cruise-automation/rosbag.js/blob/bc264c7fe1688e35a4c63dd9654b7a582e271b24/src/bag.js#L111-L121 As I pointed out above, the C++ bag reader handles this by keeping a separate iterator for each chunk and sorting them after each message is emitted. https://github.com/ros/ros_comm/blob/29053c4832229efa7160fb944c05e3bc82e11540/tools/rosbag_storage/src/view.cpp#L151