sftp fails to connect with pysftp 0.2.9

Expected behaviour:

successful sftp connection with latest version of pysftp, 0.2.9.

Actual behaviour:

fails to connect. works on pysftp 0.2.8.

Steps to reproduce:

• Step 1: create ssh key: ssh-keygen -t rsa -b 2048
• Step 2: flexget --loglevel debug execute sync

Config:

sync:
  sftp_list:
    host: '{? sftp.host ?}'
    port: '{? sftp.port ?}'
    username: '{? sftp.username ?}'
    private_key: '{? sftp.private_key ?}'
    dirs:
      - '{? remote.downloads ?}/'
    recursive: true
  regexp:
    accept:
      -  '.*\.mkv$'
  free_space:
    <<: *free_space
    space: 1500
  sftp_download:
    to: '{? local.downloads ?}/'

Log:

(click to expand)

2020-05-26 08:28:04 DEBUG    sftp_client   sync         Caught exception: No hostkey for host $IP_ADDRESS found.
2020-05-26 08:28:04 WARNING  sftp_client   sync         Failed to connect to $IP_ADDRESS; waiting 15 seconds before retrying.

Additional information:

• FlexGet version: 3.1.57
• Python version: 3.6.9
• Installation method: virtualenv
• Using daemon (yes/no): yes
• OS and version: Ubuntu 18.04.4
• Other info: hostkey can be ignored or set to a custom file with CnOpts attributes in pysftp connection object according to the docs. it defaults to ~/.ssh/known_hosts. I’ve manually ssh’d from the flexget server to my target server before so it should have already cached the host key in known_hosts

[ksurl]

I had some time so I dove back into this issue for pysftp 0.2.9 compatibility.

Here is the gist of it: 0.2.9 requires more strict hostkey checking when connecting. If you run as a regular user with a ~/.ssh/known_hosts file it will work fine out of the box (if the hostkey is already added, which you can manually add with ssh-keyscan -H -p PORT HOSTNAME >> ~/.ssh/known_hosts). If you run as a user without a home folder (such as in docker or service account that does not have .ssh folder such as in /var/lib/flexget), you can specify a path for the known_hosts file such as

cnopts = pysftp.CnOpts(knownhosts='/path/known_hosts')

and include cnopts in the arguments for pysftp.Connection such as

sftp = pysftp.Connnection(host='host', port='port', user='username', password='password', cnopts=cnopts)

you can also keep the behavior the same as 0.2.8 (no host key checking) by setting it explicitly:

cnopts = pysftp.CnOpts() cnopts.hostkeys = None

This works fine except you will get a warning on each phase that calls a sftp connection (sftp_list, sftp_download, sftp_upload) if it can’t find ~/.ssh/known_hosts. I can’t find a way to set it to None while creating the object to suppress the warning.

I tried to take a stab at updating sftp_client.py but it is a bit over my head.

Here is what I was thinking in terms of priority:

• Handle the various locations those known_hosts could be on Windows, macOS, and Linux paths (c:\users\username, /Users/username, and /home/username (need to check if it doesn’t exist to go to 2nd and 3rd options)
• Fall back on the config directory as a secondary path to look
• Or disable by setting hostkeys to None

pseudocode:

if not exist ~/.ssh/known_hosts
  if not exist $configpath/known_hosts
    cnopts = pysftp.CnOpts=()
    cnopts.hostkeys = None
  else
    cnopts = pysftp.CnOpts(knownhosts=$configpath/known_hosts)
else
  cnopts = pysftp.CnOpts()

sftp = pysftp.Connection(blah blah, cnopts=cnopts)

 

[4oo4]

@ksurl I could never get pysftp to work with a password protected private key, if I remember correctly I just had to give it the decrypted key. I think it relates to the paramiko library that pysftp uses behind the scenes, in my experience it’s always been supper buggy.