s3fs does not work with aiobotocore in dask-cloudprovider

Hi,

I am trying to run the following code:

from dask.distributed import Client
from dask_cloudprovider import FargateCluster
from dask import dataframe as df

if __name__ == "__main__":
cluster = FargateCluster()
cluster.adapt()
client = Client(cluster)

HOLDINGS = df.read_parquet(HOLDINGS_URL)

HOLDINGS_URL is my personal s3 bucket, and this code works without using the FargateCluster.

When i do use it, I get the following error:

❯ python main.py
/home/ubuntu/miniconda3/envs/dev/lib/python3.7/contextlib.py:119: UserWarning: Creating your cluster is taking a surprisingly long time. This is likely due to pending resources on AWS. Hang tight!
  next(self.gen)
Traceback (most recent call last):
  File "main.py", line 81, in <module>
    HOLDINGS = df.read_parquet(HOLDINGS_URL)
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py", line 224, in read_parquet
    **kwargs
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/dask/dataframe/io/parquet/fastparquet.py", line 178, in read_metadata
    fs, paths, gather_statistics, **kwargs
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/dask/dataframe/io/parquet/fastparquet.py", line 119, in _determine_pf_parts
    elif fs.isdir(paths[0]):
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/s3fs/core.py", line 511, in isdir
    return bool(self._lsdir(path))
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/s3fs/core.py", line 334, in _lsdir
    for i in it:
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/aiobotocore/paginate.py", line 14, in __iter__
    "{self} is an AsyncIterable: use `async for`".format(self=self)
NotImplementedError: <aiobotocore.paginate.AioPageIterator object at 0x7fd87c7b3610> is an AsyncIterable: use `async for`
distributed.client - ERROR - Failed to reconnect to scheduler after 10.00 seconds, closing client
distributed.utils - ERROR -
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/distributed/utils.py", line 663, in log_errors
    yield
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/distributed/client.py", line 1296, in _close
    await gen.with_timeout(timedelta(seconds=2), list(coroutines))
concurrent.futures._base.CancelledError
distributed.utils - ERROR -
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/distributed/utils.py", line 663, in log_errors
    yield
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/distributed/client.py", line 1025, in _reconnect
    await self._close()
  File "/home/ubuntu/miniconda3/envs/dev/lib/python3.7/site-packages/distributed/client.py", line 1296, in _close
    await gen.with_timeout(timedelta(seconds=2), list(coroutines))
concurrent.futures._base.CancelledError

It seems like s3fs doesn’t have support for aiobotocore, unless I am reading this wrong. Does this make these libraries incompatible?

[jacobtomlinson]

I was curious to see how much effort it would be to switch to botocore and so have raised #277 as a POC.

[gvelchuru]

#277 works for me!