ssh filesystem and put_file are incompatible

Using a ssh filesystem and put_files gives an error

import fsspec
import sshfs
fs = fsspec.filesystem("ssh", host='xx.xx.xx.xx', username='eouser') 
fs.put_file(lpath="t", rpath="/home/me/t")

I receive an exception:

File "/home/me/miniconda3/envs/olci/lib/python3.9/site-packages/fsspec/callbacks.py", line 66, in relative_update
    self.value += inc
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

The reason is because the ‘write’ method in paramiko does not return the length of written data. https://github.com/jbouse-debian/paramiko/blob/master/paramiko/file.py#L385

Unless paramiko changes their function, the method ‘put_file’ in spec.py should test segment_len to be None and skip the update… or use len(data) as a default as follows:

                    segment_len = f2.write(data)
                    if segment_len is None:
                          segment_len = len(data)
                    callback.relative_update(segment_len)

I’m not sure if this is a good solution…

[martindurant]

In https://github.com/conda-forge/sshfs-feedstock/blob/main/recipe/meta.yaml , missing noarch: python in the build section.

[ghislainp]

I suppose you could try temporarily with

import fsspec
import sshfs
fsspec.register_implementation("ssh", sshfs.SSHFileSystem)

This does not work well with prefect for two reasons:

1- it seems that prefect uses the full uri for the destination path in put_file, I don’t know if it is valid but anyway the SFTP implementation was able to remove the protocle and host part. The SSHFileSystem implementation does not remove it and it fails. I don’t have a direct way to change this in prefect.

2- The SFTP implementation automatically create the required directory (e.g. uri ssh://host/directory/subdirectory/file.txt works even if directory does not exist yet). The SSHFileSystem does not create these directories. Prefect relies on the first behavior.

Let me know if these problemes are considered as bugs in the SSHFileSystem or if I should report the issue to prefect.