ImportError: cannot import name 'setup_logging' from 'fsspec.utils'

Description

I’d like to integrate SageMaker into our Kedro pipelines. To do so, I’ve tried running through the official guides on the AWS Open Source Blog tutorial and latest Kedro docs. They’re pretty similar with the major difference being that the former enforces Kedro version 0.16.5 and the latter 0.17.1. Both examples gave me the error of ImportError: cannot import name 'setup_logging' from 'fsspec.utils' (/opt/conda/lib/python3.7/site-packages/fsspec/utils.py) when executing the final pipeline via kedro run--env sagemaker.

I’ve narrowed this down to dependency issues between kedro, s3fs, and fsspec. To resolve it people have recommended updating fsspec to at least 0.9.0. That fixes it, but the problem is that kedro 0.16.5 has requirement fsspec<0.7.0,>=0.5.1 and kedro 0.17.4 has requirement fsspec<0.9,>=0.5.1. In short, it looks like Kedro’s maximum required fsspec version would have to be bumped beyond 0.9.0 for all three libraries to be compatible and for the integration with SageMaker to work properly.

Context

This prevents me from integrating Kedro and SageMaker. The version of fsspec needed to resolve the ImportError: cannot import name 'setup_logging' from 'fsspec.utils' error is not compatible with the latest Kedro release.

Steps to Reproduce

1. Install latest version of Kedro (currently 0.17.4) and the latest compatible versions of s3fs and fsspec. Version of Kedro doesn’t matter here since restriction is coming from fsspec being limited below 0.9.0. !pip install -qU s3fs==0.5.0 fsspec==0.8.7 kedro==0.17.4
2. Import libraries and force the error

import s3fs
import fsspec
import kedro
print(s3fs.__version__)
print(fsspec.__version__)
print(kedro.__version__)

from fsspec.utils import setup_logging

Expected Result

In fsspec versions greater than 0.9.0 we don’t receive the ImportError: cannot import name 'setup_logging' from 'fsspec.utils' error. This would allow us to execute the SageMaker tutorials end-to-end without issue.

Actual Result

We receive the following error:

0.5.0
0.8.7
0.17.4
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-73c147de1a54> in <module>()
      6 print(kedro.__version__)
      7 
----> 8 from fsspec.utils import setup_logging

ImportError: cannot import name 'setup_logging' from 'fsspec.utils' (/usr/local/lib/python3.7/dist-packages/fsspec/utils.py)

This is the same error that appears when executing the tutorial pipelines, just providing a condensed version here that zeroes in on the problem.

Your Environment

• I’ve tested multiple versions of Kedro including 0.16.5 / 0.17.1 / 0.17.4
• Python 3.7.10
• Tested in SageMaker Studio and Google Colab

@datajoely - Tagging you here re: our chat. Hope this additional information helps and thank you once again!

[ignacioparicio]

We have upgraded our fsspec requirement from fsspec>=0.5.1, <0.9 to fsspec>=2021.04, <2022.01 (the upper bound to be revised past that date). The change will come into effect with the next Kedro release, be it 0.17.5 or 0.18.0.

I’m closing this issue for now, but please shout if you want it reopen.

[datajoely]

@ecrows what other packages are conflicting here? Also, what version of Kedro are you working with?