Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Allow saving to compressed files
See original GitHub issueIs there any way to use gzip (or bzip2 or xz) compression when saving a pandas DataFrame to a feather file? Something like feather.write_dataframe(df, path, compression='gzip'), the way you can do with DataFrame.to_csv(...)?
I work a lot with files stored on an NFS network drive, and feather.write_dataframe stresses our network and creates files which are much larger than the equivalent gzip-compressed CSVs. I tried running gzip -9 on a feather file and got ~10x reduction in size.
I know compression has been mentioned already (#24, #76), but those seem to deal with changing the feather format to support block compression. I just want to gzip the entire file at the same time as I am saving it (and not afterwards using gzip).
Thank you!
Issue Analytics
- State:
- Created 7 years ago
- Reactions:1
- Comments:14 (9 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
hi @fbrundu we’ve set up the compression toolchain within Apache Arrow and I’m standing by to add compression to the Feather format as soon as we are able to ship R bindings for the Arrow C++ libraries (and port over the data.frame conversion code from this repo there as part of the R-Arrow interop layer)
This is done in Feather V2 (lz4 and zstd compression available) coming in Arrow 0.17.0