Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

EOFError with Gzip read

See original GitHub issue

I created an issue on the Pandas repo, but it looks like it might be an s3fs, versions 0.3.0 and greater, error.

import pandas as pd
data = pd.read_csv("s3://bucketname/file.csv.gz")

Gives the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/site-packages/pandas/io/parsers.py", line 463, in _read
    data = parser.read(nrows)
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/site-packages/pandas/io/parsers.py", line 1154, in read
    ret = self._engine.read(nrows)
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/site-packages/pandas/io/parsers.py", line 2059, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2124, in pandas._libs.parsers.raise_parser_error
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/home/ubuntu/miniconda3/envs/test/lib/python3.7/gzip.py", line 482, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

Setup

Installed Pandas and s3fs via pip:

pip install pandas s3fs

Pandas version: 0.25.1 s3fs version: 0.3.3

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:16 (6 by maintainers)

Top GitHub Comments

3reactions

TomAugspurgercommented, Aug 29, 2019

I’m able to reproduce, but only on a real request against s3, not with moto. I’ll look into this today though, and should be able to do a release of fsspec once it’s fixed.

1reaction

TomAugspurgercommented, Aug 29, 2019

0.3.4 is on PyPI. Will show up on Conda-forge later today.

Top Results From Across the Web

Compressed file ended before the end-of-stream marker was ...

I have a big file and I'm trying to read and rewrite it. But some lines in the file seem to be corrupted...

gzip — Support for gzip files — Python 3.11.1 documentation

This module provides a simple interface to compress and decompress files just like ... EOFError and zlib.error can also be raised for invalid...

1648621 – [abrt] lutris: read(): gzip.py:482:read:EOFError

Bug 1648621 - [abrt] lutris: read(): gzip.py:482:read:EOFError: Compressed file ended before the end-of-stream marker was reached.

[traceback] "EOFError: Compressed file ended before the end ...

py", line 276, in read return self._buffer.read(size) File "python3.6/gzip.py", line 482, in read raise EOFError("Compressed file ended before the " """ We ...

EOFError: Compressed file ended before the ... - QIIME 2 Forum

File “/home/213002h/miniconda3/envs/qiime2-2018.4/lib/python3.5/gzip.py”, line 480, in read raise EOFError("Compressed file ended before the ...