Remote connection via ssh gets stuck on Linux

I want to create a console using a remote kernel through ssh to tap into the power of the cloud. For this, I am using the command: jupyter qtconsole --existing /path/to/remotemachine.json --ssh user@ip -JupyterQtConsoleApp.sshkey=/path/to/secret

On my win10 laptop running an anaconda installation, this works flawlessly and I can also use the dialog in spyder to connect as well. Very cool. However, on my Linux box (Arch with an anaconda environment installed), the same command gets stuck (and using the spyder dialog completely crashes spyder). When running the same command as above with --debug, I get:

[JupyterQtConsoleApp] Searching ['/home/xxx/git/project1', '/home/xxx/.jupyter', '/home/xxx/anaconda3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[JupyterQtConsoleApp] Looking for jupyter_config in /etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_config in /usr/local/etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_config in /home/xxx/anaconda3/etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_config in /home/xxx/.jupyter
[JupyterQtConsoleApp] Looking for jupyter_config in /home/xxx/git/numerai
[JupyterQtConsoleApp] Looking for jupyter_qtconsole_config in /etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_qtconsole_config in /usr/local/etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_qtconsole_config in /home/xxx/anaconda3/etc/jupyter
[JupyterQtConsoleApp] Looking for jupyter_qtconsole_config in /home/xxx/.jupyter
[JupyterQtConsoleApp] Looking for jupyter_qtconsole_config in /home/xxx/git/numerai
[JupyterQtConsoleApp] Connecting to existing kernel: /home/xxx/.ssh/remotemachine.json
[JupyterQtConsoleApp] Loading connection file /home/xxx/.ssh/remotemachine.json
[JupyterQtConsoleApp] Loading connection file /home/xxx/.ssh/remotemachine.json
[JupyterQtConsoleApp] Forwarding connections to 127.0.0.1 via user@ip

and then nothing happens anymore. On Win10, the next thing is I get prompted for the key’s passphrase, but on Linux: nothing. When I CTRL-C out, I get the following traceback:

^C[JupyterQtConsoleApp] ERROR | Could not setup tunnels                                                                      
Traceback (most recent call last):                                                                                           
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/jupyter_client/consoleapp.py", line 237, in init_ssh              
    newports = tunnel_to_kernel(info, self.sshserver, self.sshkey)                                                           
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/jupyter_client/connect.py", line 264, in tunnel_to_kernel         
    if tunnel.try_passwordless_ssh(sshserver, sshkey):                                                                       
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/jupyter_client/ssh/tunnel.py", line 73, in try_passwordless_ssh   
    return f(server, keyfile)                                                                                                
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/jupyter_client/ssh/tunnel.py", line 93, in _try_passwordless_openssh                                                                                                                           
    i = p.expect([ssh_newkey, _password_pat], timeout=.1)                                                                    
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/spawnbase.py", line 343, in expect                        
    return self.expect_list(compiled_pattern_list,                                                                           
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/spawnbase.py", line 372, in expect_list                   
    return exp.expect_loop(timeout)
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/expect.py", line 169, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/pty_spawn.py", line 500, in read_nonblocking
    if (timeout != 0) and select(timeout):
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/pty_spawn.py", line 450, in select
    return select_ignore_interrupts([self.child_fd], [], [], timeout)[0]
  File "/home/xxx/anaconda3/lib/python3.8/site-packages/pexpect/utils.py", line 143, in select_ignore_interrupts
    return select.select(iwtd, owtd, ewtd, timeout)
KeyboardInterrupt
[JupyterQtConsoleApp] Exiting application: jupyter-qtconsole

interestingly, after the ctrl-c, there is an error in front of the traceback. No idea if that means the tunnel could not get up or just a consequence from the Interrupt. I should note that I can establish a ssh connection on my Linux box to the remote no problem. Running the ssh -f -S none -L 127.0.0.1:port1:127.0.0.1:port2 user@ip -i /path/to/secret sleep 60 manually to establish tunnels works flawlessly too. So my key pair should be good.

My anaconda3 installation is on python 3.8, 64bit and the following versions:

jupyter core     : 4.7.1
jupyter-notebook : 6.3.0
qtconsole        : 5.0.3
ipython          : 7.22.0
ipykernel        : 5.3.4
jupyter client   : 6.1.12
jupyter lab      : 3.0.11
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.1.3
traitlets        : 5.0.5

paramiko-2.7.2 is installed.

I am not a github ninja by any means, so please let me know if further info is required.

[green-slowmoe]

got it! this is a duplicate of jupyter/jupyter_client#494 my linux box asks for Enter passphrase for key '/path/to/secret': so the regex 'pass(word|phrase):' fails because the column is a bit later than expected. Removed column in regex, works.

Edit: celbrated a bit prematurely. Opening a standalone qtconsole works now, but the spyder integrated one was still broken. Tracked this down to this, same problem with the column (and also word!=phrase). Removed it, now spyder does not crash but the console still won’t open. Guess I make an issue over there?

Edit2: nevermind, the console opens in spyder after the fix to this file as well. Hooray!

[ccordoba12]

Ok, and what’s the fix that we need to do here?