Pandas pd.read_csv(s3_path) fails with “TypeError: 'coroutine' object is not subscriptable”

What happened: Since the latest version of Pandas uses s3fs underneath in order to read files from S3 buckets, the latest release of s3fs causes errors in doing so. Calling the read_csv function generates TypeError: 'coroutine' object is not subscriptable error.

What you expected to happen: It should have read the CSV file from S3 without any issues.

Minimal Complete Verifiable Example:

import pandas as pd
df = pd.read_csv(s3_path)

Error Log:

Traceback (most recent call last):
  File "spark.py", line 84, in <module>
    df=pd.read_csv('s3://<bucketname>/<filename>.csv')
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/parsers.py", line 686, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/parsers.py", line 435, in _read
    filepath_or_buffer, encoding, compression
  File "/usr/local/lib64/python3.7/site-packages/pandas/io/common.py", line 222, in get_filepath_or_buffer
    filepath_or_buffer, mode=mode or "rb", **(storage_options or {})
  File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 133, in open
    out = self.__enter__()
  File "/usr/local/lib/python3.7/site-packages/fsspec/core.py", line 101, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/usr/local/lib/python3.7/site-packages/fsspec/spec.py", line 844, in open
    **kwargs
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 394, in _open
    autocommit=autocommit, requester_pays=requester_pays)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 1276, in __init__
    cache_type=cache_type)
  File "/usr/local/lib/python3.7/site-packages/fsspec/spec.py", line 1134, in __init__
    self.details = fs.info(path)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 719, in info
    return sync(self.loop, self._info, path, bucket, key, kwargs, version_id)
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 51, in sync
    raise exc.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 35, in f
    result[0] = await future
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 660, in _info
    Key=key, **version_id_kw(version_id), **self.req_kw)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 214, in _call_s3
    raise translate_boto_error(err)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 207, in _call_s3
    return await method(**additional_kwargs)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 121, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 140, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 90, in _send_request
    exception):
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 199, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 1225, in redirect_from_error
    new_region = self.get_bucket_region(bucket, response)
  File "/usr/local/lib/python3.7/site-packages/botocore/utils.py", line 1283, in get_bucket_region
    headers = response['ResponseMetadata']['HTTPHeaders']
TypeError: 'coroutine' object is not subscriptable
sys:1: RuntimeWarning: coroutine 'AioBaseClient._make_api_call' was never awaited

Anything else we need to know?: Running python3 -m pip install s3fs==0.4.2 instead of the latest release resolves the above issue.

Environment:

• Python version: 3.7.6
• Operating System: Amazon Linux 2
• Install method (conda, pip, source): pip python3 -m pip install s3fs

[martindurant]

The following diff appears to show that the situation is normally a “bad request”, where botocore has not been able to determine the appropriate region for the bucket, which, in turn, is likely a credentials problem (e.g., expired token)

--- a/s3fs/core.py
+++ b/s3fs/core.py
@@ -211,7 +211,17 @@ class S3FileSystem(AsyncFileSystem):
             except Exception as e:
                 err = e
                 break
-        raise translate_boto_error(err)
+        if "'coroutine'" in str(err):
+            # aiobotocore internal error - fetch original botocore error
+            tb = err.__traceback__
+            while tb.tb_next:
+                tb = tb.tb_next
+            try:
+                await tb.tb_frame.f_locals['response']
+            except Exception as e:
+                err = e
+        raise translate_boto_error(err) from err

     call_s3 = sync_wrapper(_call_s3)

cc @chinmaychandak

[martindurant]

We are working on this. You can try setting the environment variable AWS_DEFAULT_REGION as a workaround.