Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot do chunked reads on compressed files. To read, set blocksize=None

See original GitHub issue

I tried to read data from compressed file to dask bag as follows:

bag = db.read_text('[path/to/gzip/file/data.gz]', blocksize="2.5GB").str.strip().str.split('\t')

but I got error: Cannot do chunked reads on compressed files. To read, set blocksize=None

If I changed blocksize to be None as suggested, the bag only had 1 partition, which was dumped to 1 worker for further computation no matter how many workers were available.

I also tried bag.repartition(npartitions=6), but all works were still assigned to 1 worker (even though there were 2 workers available).

How to read data from one gzip file and have multiple partitions which can be assigned to multiple workers for computation?

Issue Analytics

State:
Created 2 years ago
Comments:9 (5 by maintainers)

Top GitHub Comments

1reaction

mrocklincommented, Jun 8, 2021

Technically you could probably use lzma / xz files. But that’s somewhat exotic. Pile-of-gzip for the win.

On Tue, Jun 8, 2021 at 2:34 PM Fred Li @.***> wrote:

yes, the gzip file contains tsv format files that cannot be changed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask/issues/7758#issuecomment-857050065, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTHDROXZSVKKAJLDA4TTRZWDTANCNFSM46ADYURA .

0reactions

fredmscommented, Jun 8, 2021

yes, the gzip file contains tsv format files that cannot be changed.