Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Azure blob requires fsspec >= 0.8 and kedro is still on 0.7

See original GitHub issue

Description

Azure blob with authentication via Shared Access Signature requires fsspec >= 0.8

Context

Kedro install does not compile, kedro viz does not work with fsspec 0.8

Steps to Reproduce

pip install -U fsspec
pip install adlfs
kedro run ( with azure blob dataset)

Expected Result

load data from Azure blob storage

Actual Result

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/lucianoissoe/opt/anaconda3/lib/python3.7/site-packages/adlfs/__init__.py", line 1, in <module>
    from .spec import AzureDatalakeFileSystem
  File "/Users/lucianoissoe/opt/anaconda3/lib/python3.7/site-packages/adlfs/spec.py", line 21, in <module>
    from fsspec.asyn import (
ModuleNotFoundError: No module named 'fsspec.asyn'

kedro.io.core.DataSetError: 
No module named 'fsspec.asyn'.
Failed to instantiate DataSet 'safras' of type `kedro.extras.datasets.pandas.csv_dataset.CSVDataSet`.
Error: 
No module named 'fsspec.asyn'.
Failed to instantiate DataSet 'safras' of type `kedro.extras.datasets.pandas.csv_dataset.CSVDataSet`.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

Kedro version used (pip show kedro or kedro -V):kedro, version 0.16.6
Python version used (python -V):Python 3.7.6
Operating system and version: Mac

Issue Analytics

State:
Created 3 years ago
Reactions:3
Comments:5 (4 by maintainers)

Top GitHub Comments

1reaction

lorenabalancommented, Nov 16, 2020

Hi @Lucianois thanks for raising this. We are aware this is really inconvenient, but it will be made available in version 0.17.0. Commit https://github.com/quantumblacklabs/kedro/commit/151cbcd1acaae25da4633ab87fabbf354d60059b was pushed to develop branch.

1reaction

Minyuscommented, Nov 4, 2020

I wouldn’t say this is a duplicate.

Despite the #489 comment, as of 4th Nov 2020, https://github.com/quantumblacklabs/kedro/blob/develop/requirements.txt#L5 sets fsspec>=0.5.1, <0.8, which means that current develop branch does not support fsspec >= 0.8 and cannot solve this Azure blob issue.

PyTorch Lightning also requires fsspec>=0.8 and got the same problem: https://github.com/quantumblacklabs/kedro/issues/553