SSH: ValueError: Can't create any SFTP connections!

Bug Report

I open this issue as a follow up to https://github.com/iterative/dvc/issues/6138

Description

dvc push raises an error when trying to push to an SFTP remote. It used to work with older versions. The SFTP remote I use is a personal Raspberry Pi server. I did not change anything on the server.

Reproduce

Unfortunately, since I use a private server, I don’t know whether it would be easy to reproduce.

After updating dvc, I tried to run dvc push and I got these logs:

$ dvc push -v -a
2021-10-03 11:59:51,299 DEBUG: Preparing to transfer data from '.dvc/cache' to 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,300 DEBUG: Preparing to collect status from 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,318 DEBUG: Collecting status from 'ssh://<REMOTE URL>/gcoter/music-generation-v2.dvc'
2021-10-03 11:59:51,918 DEBUG: Querying 38 hashes via object_exists
2021-10-03 12:00:30,881 ERROR: unexpected error - Can't create any SFTP connections!                                                                                         
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 57, in run
    processed_files_count = self.repo.push(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/repo/push.py", line 48, in push
    pushed += self.cloud.push(obj_ids, jobs, remote=remote, odb=odb)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/data_cloud.py", line 85, in push
    return transfer(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 221, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 160, in compare_status
    dest_exists, dest_missing = status(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 420, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 411, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 96, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/fsspec/asyn.py", line 480, in _exists
    await self._info(path)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/spec.py", line 135, in _info
    async with self._pool.get() as channel:
  File "/usr/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/home/gcoter/.cache/pypoetry/virtualenvs/music-generation-v2-7Nqe6-3y-py3.8/lib/python3.8/site-packages/sshfs/pools/soft.py", line 38, in get
    raise ValueError("Can't create any SFTP connections!")
ValueError: Can't create any SFTP connections!
------------------------------------------------------------
2021-10-03 12:00:31,971 DEBUG: Version info for developers:
DVC version: 2.7.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-88-generic-x86_64-with-glibc2.29
Supports:
	http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
	ssh (sshfs = 2021.9.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda5
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda5
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-10-03 12:00:31,973 DEBUG: Analytics is enabled.
2021-10-03 12:00:32,040 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpkmytybwl']'
2021-10-03 12:00:32,043 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpkmytybwl']'

Expected

Since I did not change the configuration of my server and it used to work, I would expect dvc push to work.

Environment information

$ dvc doctor
DVC version: 2.7.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-89-generic-x86_64-with-glibc2.29
Supports:
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
        ssh (sshfs = 2021.9.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda5
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda5
Repo: dvc, git

[ilankor]

Thanks! The strange thing is, other users can run “dvc push/pull” from their profiles and the same server

[dberenbaum]

@dtrifiro You might want to keep an eye on this one.