Avoid Kedro fsspec requirements being mutually incompatible with pandas 1.1.0

Description

Kedro 16.4 enforces fsspec<0.7.0,>=0.5.1 & PANDAS = "pandas>=0.24, <1.0.4" in setup.py

However,

1. the pd upper version limit is not enforced, so newer pd versions can coexist after pip install kedro
2. The blocking issue (for kedro including higher pd versions) on parquet reads in pandas is resolved re: https://github.com/pandas-dev/pandas/issues/34467 in pd >1.0.5, so the key blocker on Kedro upgrading pandas versions no longer exists.

IF kedro allows pandas 1.1.0, then you are going to hit an incompatibility with fsspec, as pandas 1.1.0 requires fsspec>=0.7.4.

Context

You cannot run kedro and pandas>1.1.0 on the same environment. Pandas needs fsspec>0.7.4.

There are meaningful improvements in newer pandas, so I would like to be able to run them together out-of-the-box.

The current reason to not allow higher pd versions as per setup.py in the kedro source code (https://github.com/pandas-dev/pandas/issues/34467) is no longer applicable, so i think it is time to make this change unless fsspec has deal-breaking problems in later versions.

Possible Implementation

Can you just update fsspec version requirements to fsspec<0.7.4,>=0.5.1? I’m not aware of any major problems in the newer versions

Possible Alternatives

Raise warnings or errors if kedro co-exists with pd>=1.1.0

[tamsanh]

Oops. FYI, I’m running into an incompatibility with s3fs 0.5.0 as it requires fsspec>=0.8.0.

[JMBurley]

Thanks both,

Would I be correct in understanding the official position here as: “all file loads should be handled by .yml in your catalog folder; it is anti-pattern to directly read files with pandas, therefore we don’t view it as critical for kedro to facilitate pandas s3 functionality, nor maintain it if it is incidentally present in some kedro versions”?

I understand the position if so, although I think there are good reasons to prototype eda in a project using pandas s3 functionality rather than hopping in and out of the data catalog to create potentially disposable items.

Regardless, looking forward to v0.17.0!

PS. I agree that anyone who doesn’t use virtual environments deserves the problems they encounter