Error reading parquet files made by AWS Athena

I made a bunch of parquet files using an amazon athena CTAS query. I downloaded these files to first test locally (the end goal is to access the data from S3).

If I run the code below;

import s3fs
from petastorm.reader import make_batch_reader
from petastorm.tf_utils import make_petastorm_dataset

dataset_url = "file:///Data/test-parquet"

with make_batch_reader(dataset_url) as reader:
    dataset = make_petastorm_dataset(reader)
    for batch in dataset:
        break
batch.correct

I receive a lot of warnings and then an error in for batch in dataset

pyarrow.lib.ArrowIOError: The file only has 1 row groups, requested metadata for row group: 1

If 1 look at dataset.take(1) or something alike, I do see the correct schema of the table. However, I don’t seem to be able to access the data.

[selitvin]

Thank you. This is a serious bug. I really appreciate you looking into this. I am in the middle of 0.8.1 release and will add this fix to the release. If you don’t mind, I’ll add you to the reviewers of the fix.

[jeisinge]

I’ve run a couple of tests and it appears to be working. I’m continuing to test with a full train and I’ll report back by the end of the week once it has run.