Logic bug in arrow_writer?

https://github.com/huggingface/datasets/blob/88a902d6474fae8d793542d57a4f3b0d187f3c5b/src/datasets/arrow_writer.py#L475-L488

I got some error, and I found it’s caused by batch_examples being {}. I wonder if the code should be as follows:

-        if batch_examples and len(next(iter(batch_examples.values()))) == 0:
+       if not batch_examples or len(next(iter(batch_examples.values()))) == 0:
            return

@lhoestq

[lhoestq]

But wouldn’t it be nice if the code can ignore it, like it ignores {“a”: []}?

I think it would make things confusing because it doesn’t follow our definition of a batch: “the columns of a batch = the keys of the dict”. It would probably break certain behaviors as well. For example if you remove all the columns of a dataset (using .remove_colums(...) or .map(..., remove_columns=...)), the writer has to write 0 columns, and currently the only way to tell the writer to do so using write_batch is to pass {}.

The error says something like arrays and schema doesn’t have the same length. And it’s not very clear I passed a {}.

Yea the message can actually be improved indeed, it’s definitely not clear. Maybe we can add a line right before the call pa.Table.from_arrays to make sure the keys of the batch match the field names of the schema

[cccntu]

Thanks, I added a if-print and I found it does return an empty examples in the chunking function that is passed to .map().