Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Write multiple parquet files from a single dataframe defined by max file size
See original GitHub issueThe Problem
I was unsuccessful in finding a way to write a single dataframe to multiple parquet files using the s3.to_parquet() method. Currently, it seems to write one parquet file which could slow down Athena queries.
Possible Solution
It would be great to have an option for the “max parquet file size” when using s3.to_parquet(). So instead of creating one large parquet file, we could create many smaller parquet files that would help optimize Athena queries.
My reasoning I believe Athena, behind the scenes, uses the number of files to split the query workload across different nodes.
Issue Analytics
- State:
- Created 3 years ago
- Comments:5 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Can I work on this particular request ?
@rparthas yep, it would be very welcome! Just make sure to checkout from our dev branch and then open a pull request against the same.