Dataset Viewer issue for darragh/demo_data_raw3

Link

https://huggingface.co/datasets/darragh/demo_data_raw3

Description

Exception:     ValueError
Message:       Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.

reported by @NielsRogge

Owner

No

[severo]

do you have an idea of why it can occur @huggingface/datasets? The dataset consists of a single parquet file.

[albertvillanova]

Apparently, there is something weird with that Parquet file: its schema is:

images: extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>>

I have forced a right schema:

from datasets import Features, Image, load_dataset

features = Features({"images": Image()})
ds = datasets.load_dataset("parquet", split="train", data_files="train-00000-of-00001.parquet", features=features)

and then recreated a new Parquet file:

ds.to_parquet("train.parquet")

Now this Parquet file has the right schema:

images: struct<bytes: binary, path: string>
  child 0, bytes: binary
  child 1, path: string

and can be loaded normally:

In [26]: ds = load_dataset("parquet", split="train", data_files="dataset.parquet")
n [27]: ds
Out[27]: 
Dataset({
    features: ['images'],
    num_rows: 20
})