[BUG] Reading data from GCS creates issue

Describe the bug Reading parquet file from Google Cloud Storage does not work.

Steps/Code to reproduce bug

dataset = nvt.Dataset("gs://bucket/file.parquet")
dataset.to_ddf().head()

Error:

cuDF failure at: ../src/table/table.cpp:42: Column size mismatch:

If the data is copied to the local disk, the code will work. cuDF / dask_cudf can read from GCS. This is with the latest NVTabular

[rjzamora]

I’m sorry for the delay here. This is indeed a bug in the optimized data-transfer logic for read_parquet from remote storage. It turns out that that the list column name is modified from “genres” to “genres.list.element” in the parquet metadata, and so we fail to transfer the data for that column. In the near future, all this logic will live directly in fsspec (and will be removed from NVTabular), but I will submit a temporary fix asap for NVT.

[pchandar]

@rjzamora sorry it took a while. It was a bit tricky to reproduce this on a test dataset. But if you copy the transformed parquet from this (Cell 18) example to a GCS bucket and then

ds = nvt.Dataset("gs://bucket/movielens.parquet")
ds.head()

will give the following error

RuntimeError: cuDF failure at: ../src/table/table.cpp:42: Column size mismatch: 76 != 20000076

A couple of observations: (1) this seem to happen only when the list columns exists; and (2) for sufficiently large datasets (when I tried slicing the problematic dataset it seemed to work fine). Hope this help reproduce the error at your end. Thanks