[BUG] async not working?

What happened:

As a warmup, we’ve been trying to do async fs.exists() for abfs and are unsure if async is not supported, version issue, bug on our end, etc

What you expected to happen:

Async existence check returns True / False, not an exception

Minimal Complete Verifiable Example:

fs = fsspec.filesystem('abfs', asynchronous=True, account_name='...', account_key='...')

#session = await fs.set_session()  # exn: no method defined

await fs.exists('abfs://container/folder/file')  # exn: Loop is not running

Anything else we need to know?:

Environment:

• Dask version: 2021.04.0
• fsspec: 2021.06.01
• adlfs: v0.7.3
• Python version: 3.7
• Operating System: Ubuntu
• Install method (conda, pip, source): conda (rapids 0.19) + updates

[hayesgb]

At this point in time, the asynchronous parameter introduced by fsspec will not enable adlfs to be called asynchronously (i.e. using the await keyword). Instead, operations within the package are executed asynchronously, such that traditional sync calls (i.e. fs.exists()) triggers async execution under the hood.

Adding direct async calls under the hood is on the roadmap.

[martindurant]

@lmeyerov , have you tried (note the underscore)

await fs._exists('abfs://container/folder/file')

?

This is how it is spelled in the other async implementations. In async mode, you must not call any of the methods that call sync().

Sounds like we can do this async off-main-thread, but cannot interleave filecache writes, so should enforce a global fsspec writelock on those tasks

Simpler than implementing and exposing a lock, would be to make the filecache threadsafe. It’s the updating of the metadata file that can cause (occasional) races, if someone wants to have a go. (It is also possible that two threads download the same file simultaneously, which is not critical, but would be wasteful). The same is true for blockcache, although it should be checked whether the same file can be opened from multiple threads.

I’m assuming in async operations that the loop does not need to be passed in explicitly, but I’d like to get @martindurant 's input on this.

Correct, if you have asynchronous=True, you should be using await and coroutines only, and the loop is the current running loop. By default, fsspec has its own loop in a thread. Tthe parameter is there for two cases:

• the user wants another thread/loop to run fsspec for sync calls
• you will use async mode, but the instance was created outside of the loop (not recommended!)

Yes, we have talked about the possibility to treat IO tasks differently in dask, but no concrete work yet. In some cases, IO can use significant CPU (e.g., HTTP automatic compression/encryption); what really sets IO apart is high latency. It may be reasonable for a dask thread to secede during an fsspec blocking call.