Question regarding snappy vs pyarrow snappy

Hi, For the same dataset, comparing the 2 libs I get totally different parquet file sizes. The file size from fastparquet is x4 times larger than the one given by pyarrow.

I tried changing the values of row_group_offsets and the x4 figure was the best I got. The pyarrow default is 128 MB.

For what can I attribute this vastly different file size ? How can I reduce it ?

[xhochy]

DictionaryEncoding is a way to save data in Parquet. It is simply a more efficient approach if your data not unique in each row. As far as I understand, fastparquet currently does not support writing it or does not have it enabled by default ( @martindurant correct me if I’m wrong). The implications should simply be different file sizes. Reading Parquet files with dictionary encoding is supported by all standard-compliant Parquet readers and should give you the same performance (often it does even get you a significantly better performance as file size is much smaller and this is the read speed limitation if you read e.g. over network).

[wesm]

Note that other Parquet implementations: parquet-cpp, parquet-mr, Impala, etc. always dictionary encode all column types by default.