[IDEA] Implement progress callbacks

It is a very good aspect of the UI to deliver users with updates regarding the process, if the operation is something that takes a while. There are certain operations in fsspec that are like blackboxes. You don’t know anything until those are completed. I’d propose we add a callback system for certain APIs, so that client code can show progress bars, take action depending on those etc. etc.

Here is the list of APIs that might be helpful with progress bars;

• get_file/put_file => A progress bar showing the full size of the remote file, and the current download/update status.
• get/put/cat => Total number of files that are going to be transferred, and the current cursor (like 2 of 16 files are transffered)

There are also possibilities for other APIs (e.g ls/find) to implement such callbacks, if they take a long time.

Callbacks are totally optional, and no file system is expected to implement all of them for their all APIs. Also the callback system should be abstracted to level where they are totally separate from the actual logic in the client code.

Here is the implementation that I’ve drafted (though I know it is a huge change, so it can drastically evolve): https://github.com/intake/filesystem_spec/compare/master...isidentical:progress?expand=1

Comments would be appreciated.

[isidentical]

CC: @martindurant @hayesgb @TomAugspurger @efiop

[isidentical]

Please correct me if I’m wrong, but another thing is that currently sshfs only provides an ssh filesystem, while fsspecs is pure sftp and for some use cases that might make sshfs not work

Though all operations currently supported is already using SFTP under the hood in sshfs, so there wouldn’t be any breakage on that front. Only the new methods (cp_file/checksum, and in some rare cases move() [but those would result with an error in the existing SFTPFileSystem]) does command execution.