Read Parquet directly into string[pyarrow]

I would like to read parquet data directly into string pyarrow dtypes

So I’m trying this naively on a dataset:

import dask.dataframe as dd

df = dd.read_parquet(
    "s3://nyc-tlc/trip data/fhvhv_tripdata_2022-06.parquet", 
    split_row_groups=True, 
    use_nullable_dtypes=True,
)
df.dtypes

hvfhs_license_num               object
dispatching_base_num            object
originating_base_num            object
request_datetime        datetime64[ns]
on_scene_datetime       datetime64[ns]
pickup_datetime         datetime64[ns]
dropoff_datetime        datetime64[ns]
PULocationID                     int64
DOLocationID                     int64
trip_miles                     float64
trip_time                        int64
base_passenger_fare            float64
tolls                          float64
bcf                            float64
sales_tax                      float64
congestion_surcharge           float64
airport_fee                    float64
tips                           float64
driver_pay                     float64
shared_request_flag             object
shared_match_flag               object
access_a_ride_flag              object
wav_request_flag                object
wav_match_flag                  object
dtype: object

This is especially important to me because in this case one row group is 10GB when stored naively. The data is somewhat unreadable in its current state on modest machines. I also suspect that I’m spending almost all of my time just creating and then destroying Python objects.

Any thoughts @rjzamora @ian-r-rose ?

[mroeschke]

Awesome! I put it on the meeting agenda so would be great to hear a representative pitch from the Dask side

[mrocklin]

Oh cool. And memory usage is lower in that case too.

OK, I guess then this becomes more a question of how Dask could be leveraging this as well