deploy:update connection refused

Been banging my head against the wall trying to figure out why I can’t get this working. I’m convinced this must be a bug with how ssh-pool is opening connections with the remote server. Every time I attempt to deploy, I’m met with the following error:

'deploy:update' errored after 20 s
Error: Command failed: ssh user@myhost "echo \"74ad4f770102875ded99dcc781d8c3100cf12c5a\" > /var/apps/app/releases/20180422202738/REVISION"
ssh: connect to host myhost port 22: Connection refused

at ChildProcess.exithandler (child_process.js:273:12)
at ChildProcess.emit (events.js:180:13)
at maybeClose (internal/child_process.js:936:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:220:5)

Yes, the environment is setup correctly with keys. I can ssh manually to my server (with the same user that shipit is using), and ssh’ing from the server to itself also works without any issues.

After setting the sshd log level to verbose & inspecting my server’s auth log (/var/log/auth.log), I noticed that shipit’s ssh attempts succeed, but are then immediately closed by the client (not the server):

Apr 22 20:27:38 api sshd[13276]: Accepted publickey for user from <IP_ADDRESS> port 51536 ssh2: RSA SHA256: ---
Apr 22 20:27:38 api sshd[13276]: pam_unix(sshd:session): session opened for user user by (uid=0)
Apr 22 20:27:38 api systemd-logind[1384]: New session 621 of user user.
Apr 22 20:27:38 api sshd[13308]: Received disconnect from <IP_ADDRESS> port 51536:11: disconnected by user
Apr 22 20:27:38 api sshd[13308]: Disconnected from <IP_ADDRESS> port 51536
Apr 22 20:27:38 api sshd[13276]: pam_unix(sshd:session): session closed for user user
Apr 22 20:27:38 api systemd-logind[1384]: Removed session 621.

Also, while trying to get to the source of this issue, it occurred to me that supporting ssh verbosity levels (-v, -vv, -vvv) would be an extremely useful feature. Are there any plans to add those in the future?

[evenfrost]

TL;DR: Login to your droplet and do ufw allow ssh.

So, I faced it myself and did some digging. This ‘ssh rate limit’ is because active ufw service, which is default firewall for Ubuntu. Interestingly enough, it is disabled in default Ubuntu (16.04) droplet, but enabled in some others (Ubuntu 18.04 with Docker in my case).

You can check status with ufw status, it should say something like 22/tcp LIMIT if ssh connections are rate-limited. Default hitcount (concurrent ssh connections) is 6 and can be found in /etc/ufw/user.rules, as seen here:

So, in order to fix issue with deployment, you either do ufw allow ssh to disable rate-limiting to port 22 at all or fine-tune limits (especially hitcount param seems to do the trick) to allow shipit to do its job while having some DDOS prevention mechanism as well.

More info here:

• https://serverfault.com/a/930379/520037
• https://serverfault.com/questions/368523/rate-limiting-with-ufw-setting-limits
• https://www.cyberciti.biz/faq/howto-limiting-ssh-connections-with-ufw-on-ubuntu-debian/
• http://lepepe.github.io/sysadmin/2016/01/19/ubuntu-server-ufw.html

[jonathandion]

Same problem 😕

Everything is copied successfully inside “releases” but got Connection refused before deploy:update