Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

still doesn't work with boto3 StreamingBody

See original GitHub issue

ref: https://github.com/pandas-dev/pandas/issues/16135 https://github.com/pandas-dev/pandas/pull/16150

Example

s3_object = client.get_object(Bucket=bucket, Key=key)
result = read_csv(s3_object["Body"])
# ValueError: Invalid file path or buffer object type: <class 'botocore.response.StreamingBody'>

Problem description

issue: is_file_like requirements are too strict for boto3 S3 objects https://github.com/pandas-dev/pandas/issues/16135 pull request: https://github.com/pandas-dev/pandas/pull/16150

This issue was closed but it’s not working. I’ve found that the test is skipped and has a bug. I’ll send a pull request to reproduce the behavior.

I think it’s because botocore.response.StreamingBody doesn’t have __iter__ so is_file_like returns False.

INSTALLED VERSIONS

commit: c8dcf194c7a5264ed6d9161d5807e9da972cab4c python: 2.7.10.final.0 python-bits: 64 OS: Darwin OS-release: 14.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.21.0.dev+317.gc8dcf19 pytest: 3.1.3 pip: 7.1.2 setuptools: 36.2.6 Cython: 0.26 numpy: 1.13.1 scipy: None pyarrow: None xarray: None IPython: 5.0.0 sphinx: None patsy: None dateutil: 2.6.1 pytz: 2017.2 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: None openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: 0.1.2 pandas_gbq: None pandas_datareader: None

Issue Analytics

State:
Created 6 years ago
Reactions:8
Comments:14 (10 by maintainers)

Top GitHub Comments

2reactions

gfyoungcommented, Aug 11, 2017

@uiureo : Thanks for letting us know! That would definitely be one way to resolve this issue.

2reactions

uiurcommented, Aug 11, 2017

This issue would be resolved if this PR https://github.com/boto/botocore/pull/1195 is merged at botocore.

Top Results From Across the Web

Unable to read botocore.response.StreamingBody due to ...

read() in the first place due to this error. We even tried to pickle the csv and send, but that also doesn't work....

S3 — Boto3 Docs 1.26.33 documentation - AWS

If the bucket that you're copying objects to uses the bucket owner enforced setting for S3 Object Ownership, ACLs are disabled and no...

Reading and writing files from/to Amazon S3 with Pandas

The reason is that we directly use boto3 and pandas in our code, but we won't use the s3fs directly. Still, pandas needs...

boto/boto3 - Gitter

When I run this function I get the expected response["Payload"] of <botocore.response.StreamingBody object at 0x7f32a64ac950> . So far this works as expected.

Working with really large objects in S3 - alexwlchan

import zipfile import boto3 s3 = boto3.client("s3") ... To process a ZIP file (like many formats), you don't need to load the entire...