dvc push: [Errno 111] Connect call failed

Bug Report

Description

Upon calling dvc push this error appears:

$ dvc push
ERROR: unexpected error - [Errno 111] Connect call failed (COMPANY_SERVER_IP, PORT)
                                                       
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

(I am not able to provide COMPANY_SERVER_IP, PORT due to company policy. Port differs from 22).

Reproduce

1. dvc push

Expected

An ability to push files to a company DVC server.

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 2.8.1 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-88-generic-x86_64-with-glibc2.29
Supports:
        webhdfs (fsspec = 2021.10.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        ssh (sshfs = 2021.8.1)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda2
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda2
Repo: dvc, git

Additional Information (if any):

I was able to resolve this issue by downgrading to dvc version of 2.5.2. DVC push works fine there.

[pmrowla]

@SokolovMichail @efiop

I’m unable to reproduce the issue - using the hostname and port we were provided after the discussion, I can connect to the host directly using asyncssh, sshfs, DVC’s fs.ssh.SSHFileSystem, and using status -c from a test DVC repo with SSH remote configured to use /home/dvc on the provided remote host.

I tested using the current DVC pip release (2.8.3) on both macos and linux (ubuntu-focal), and had no connection issues on either platform.

It seems like whatever the issue is, it must be related to something that is standardized across your developers’ machines/environments (since it was reproducible for multiple users on your network)? But I’m not really sure how to proceed from here.

linux results (with host:port removed):

---

the two test python scripts are:

import asyncio, asyncssh, sys

async def run_client():
    async with asyncssh.connect(HOST, port=PORT, username="dvc") as conn:
        result = await conn.run('echo "Hello!"', check=True)
        print(result.stdout, end='')
        print("connected")

try:
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

from dvc.fs.ssh import SSHFileSystem

fs = SSHFileSystem(
    host=HOST,
    port=PORT,
    username="dvc",
)
print(fs.exists("/home/dvc"))

and the test DVC repo just contains a single dvc added file foo

[SokolovMichail]

Providing dvc push -v output as requested. Due to the company policy it had to be doctored to remove the confidential information. It is replaced with <INFORMATION>.

(venv) $ dvc push -v
2021-10-17 00:25:42,632 DEBUG: Preparing to transfer data from '.dvc/cache' to 'ssh://dvc@<COMPANY_SERVER_NAME>/<PATH_TO_FOLDER>
2021-10-17 00:25:42,636 DEBUG: Preparing to collect status from 'ssh://dvc@<COMPANY_SERVER_NAME>/<PATH_TO_FOLDER>'
2021-10-17 00:25:42,637 DEBUG: Collecting status from 'ssh://dvc@<COMPANY_SERVER_NAME>/<PATH_TO_FOLDER>'
2021-10-17 00:25:42,638 DEBUG: Querying 6 hashes via object_exists
2021-10-17 00:25:48,186 ERROR: unexpected error - [Errno 111] Connect call failed ('<COMPANY_SERVER_IP>', <PORT>)                                   
------------------------------------------------------------
Traceback (most recent call last):
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/command/data_sync.py", line 57, in run
    processed_files_count = self.repo.push(
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/repo/push.py", line 48, in push
    pushed += self.cloud.push(obj_ids, jobs, remote=remote, odb=odb)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/data_cloud.py", line 85, in push
    return transfer(
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/status.py", line 160, in compare_status
    dest_exists, dest_missing = status(
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/status.py", line 36, in _indexed_dir_hashes
    indexed_dir_exists.update(odb.list_hashes_exists(indexed_dirs))
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/db/base.py", line 415, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/objects/db/base.py", line 406, in exists_with_progress
    ret = self.fs.exists(path_info)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 136, in exists
    return self.fs.exists(self._with_bucket(path_info))
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "<PATH_TO_VENV>venv/lib/python3.8/site-packages/dvc/fs/ssh.py", line 117, in fs
    return _SSHFileSystem(**self.fs_args)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/sshfs/spec.py", line 77, in __init__
    self._client, self._pool = self.connect(
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/sshfs/spec.py", line 92, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/usr/lib/python3.8/contextlib.py", line 568, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/asyncssh/misc.py", line 220, in __aenter__
    self._result = await self._coro
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/asyncssh/connection.py", line 6803, in connect
    return await _connect(options, loop, flags, conn_factory,
  File "<PATH_TO_VENV>/venv/lib/python3.8/site-packages/asyncssh/connection.py", line 297, in _connect
    _, conn = await loop.create_connection(conn_factory, host, port,
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('<COMPANY_SERVER_IP>', <PORT>)
------------------------------------------------------------
2021-10-17 00:25:49,045 DEBUG: Version info for developers:
DVC version: 2.8.1 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.4.0-88-generic-x86_64-with-glibc2.29
Supports:
        webhdfs (fsspec = 2021.10.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        ssh (sshfs = 2021.8.1)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda2
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda2
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2021-10-17 00:25:49,046 DEBUG: Analytics is enabled.
2021-10-17 00:25:49,090 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpic0pzgti']'
2021-10-17 00:25:49,092 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpic0pzgti']'