[BUG] isfile/isfolder for non-existent items takes ~50s

What happened:

Calling isile + isfolder for path abfs://c/f/2/3 when container & folder c/f/2/ exists but file 3 does not takes 25s + 25s.

What you expected to happen:

Expect to take < 1s, not 25s + 25s because calling the same az storage blob exists .... takes < 1s to get the error. This is extra surprising/negative due to asyncio support not landing yet or abfs, so it will block all other users of the python process

Minimal Complete Verifiable Example:

   %%time
    storage_options = {
        'account_name': os.environ['AZ_STORAGE_ACCOUNT_NAME'],
        'account_key': os.environ['AZ_STORAGE_ACCOUNT_KEY']    
    }
    fs = fsspec.filesystem('abfs', **storage_options)
    print(fs.isfile(path), fs.isdir(path))

Environment:

RAPIDS 0.19 stable (conda): fsspec '2021.07.0' dask ‘2021.04.0’ ubuntu

[hayesgb]

Would you mind trying the master branch? The await error is due to an error is fixed there.

On Aug 15, 2021, at 4:42 PM, lmeyerov @.***> wrote:

@hayesgb thanks!

The original issue (here) was 404s taking 30s instead of < 1s. When updating to fsspec to 2021.08.1, that seems to now be fast!

The new issue (#265) is that files that do exist now incorrectly return fs.isfile(legit_path) == False) and an async warning. I’m trying to get asynchronous=True … await fs._isfile(path) to work as a workaround, but haven’t figured that out yet. Before 2021.08.1, that worked fine in sync mode.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[lmeyerov]

Checked – the warning is gone, but wrong result + dir exists check is slowly downloading everything (afaict) instead of just checking: https://github.com/dask/adlfs/issues/265