Received disconnect: Timeout, your session not responding.

Hi,

I am trying to use asyncssh on top of python_prompt_toolkit to modernize a legacy console (colorize output, autocompletion etc.), which can be connected via SSH. I want to set an interactive input and use the same session, so that users can use the existing console with added features. The code below does the job okay, until users do not send any command (i.e, session stays idle) after a few seconds (around 20-30 seconds). Then the protocol error is received: “Timeout, your session not responding”; connection and channel are closed.

Here is the code:

#!/usr/bin/env python

import asyncio
import logging
import sys

import asyncssh
from asyncssh.connection import SSHClientConnection
from asyncssh.process import SSHClientProcess


from prompt_toolkit.shortcuts import print_formatted_text
from prompt_toolkit.shortcuts.prompt import PromptSession


logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
asyncssh.logger.set_debug_level(3)

# Alias 'print_formatted_text', so that 'print' calls go to the SSH client.
print = print_formatted_text
PROMPT = "< "


async def wait_until_banner(p: SSHClientProcess) -> str:
    out = ""
    while True:
        out = await asyncio.wait_for(p.stdout.read(65536), timeout=5)
        print(out, end='')
        if PROMPT in out:
            break
    return out


async def interact() -> None:
    prompt_session = PromptSession(message="")
    async with asyncssh.connect(host='1.2.3.4', username="foo", password="bar", keepalive_interval=5000) as conn:
        conn: SSHClientConnection
        async with conn.create_process(term_type="auto") as p:
            p: SSHClientProcess
            await wait_until_banner(p)
            # Interactive user input loop
            while True:
                out = ""
                cmd = ""
                try:
                    cmd = await asyncio.wait_for(prompt_session.prompt_async(), timeout=5)
                    p.stdin.write(f"{cmd}\r\n")
                    while True:
                        try:
                            out = await asyncio.wait_for(p.stdout.read(65536), timeout=15)
                            print(out, end='')
                            if "DESTROYING SESSION" in out and "EXIT;" in cmd:
                                return
                            elif PROMPT in out:
                                break
                        except:
                            conn.logger.debug("Timeout")
                            break
                except:
                    pass  # conn.send_packet(98) ?


try:
    asyncio.get_event_loop().run_until_complete(interact())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

Here is the output:

...snip until prompt
<  
DEBUG:asyncssh:[conn=0, chan=0, pktid=13] Received MSG_CHANNEL_REQUEST (98), 31 bytes
  00000000: 62 00 00 00 00 00 00 00 15 6b 65 65 70 61 6c 69  b........keepali
  00000010: 76 65 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 01     [e-mail].
DEBUG:asyncssh:[conn=0, chan=0] Received OpenSSH keepalive channel request
DEBUG:asyncssh:[conn=0, pktid=15] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
DEBUG:asyncssh:[conn=0, chan=0, pktid=16] Sent MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 00                                   c....
DEBUG:asyncssh:[conn=0, pktid=14] Received MSG_UNIMPLEMENTED (3), 5 bytes
  00000000: 03 00 00 00 10                                   .....
 
DEBUG:asyncssh:[conn=0, chan=0, pktid=15] Received MSG_CHANNEL_REQUEST (98), 31 bytes
  00000000: 62 00 00 00 00 00 00 00 15 6b 65 65 70 61 6c 69  b........keepali
  00000010: 76 65 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 01     [e-mail].
DEBUG:asyncssh:[conn=0, chan=0] Received OpenSSH keepalive channel request
DEBUG:asyncssh:[conn=0, pktid=17] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
DEBUG:asyncssh:[conn=0, chan=0, pktid=18] Sent MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 00                                   c....
DEBUG:asyncssh:[conn=0, pktid=16] Received MSG_UNIMPLEMENTED (3), 5 bytes
  00000000: 03 00 00 00 12                                   .....
 
DEBUG:asyncssh:[conn=0, chan=0, pktid=17] Received MSG_CHANNEL_REQUEST (98), 31 bytes
  00000000: 62 00 00 00 00 00 00 00 15 6b 65 65 70 61 6c 69  b........keepali
  00000010: 76 65 40 6f 70 65 6e 73 73 68 2e 63 6f 6d 01     [e-mail].
DEBUG:asyncssh:[conn=0, chan=0] Received OpenSSH keepalive channel request
DEBUG:asyncssh:[conn=0, pktid=19] Sent MSG_IGNORE (2), 5 bytes
  00000000: 02 00 00 00 00                                   .....
DEBUG:asyncssh:[conn=0, chan=0, pktid=20] Sent MSG_CHANNEL_SUCCESS (99), 5 bytes
  00000000: 63 00 00 00 00                                   c....
DEBUG:asyncssh:[conn=0, pktid=18] Received MSG_UNIMPLEMENTED (3), 5 bytes
  00000000: 03 00 00 00 14                                   .....
 
DEBUG:asyncssh:[conn=0, pktid=19] Received MSG_DISCONNECT (1), 50 bytes
  00000000: 01 00 00 00 02 00 00 00 25 54 69 6d 65 6f 75 74  ........%Timeout
  00000010: 2c 20 79 6f 75 72 20 73 65 73 73 69 6f 6e 20 6e  , your session n
  00000020: 6f 74 20 72 65 73 70 6f 6e 64 69 6e 67 2e 00 00  ot responding...
  00000030: 00 00                                            ..
DEBUG:asyncssh:[conn=0] Received disconnect: Timeout, your session not responding. (2)
INFO:asyncssh:[conn=0] Connection failure: Timeout, your session not responding.
INFO:asyncssh:[conn=0, chan=0] Closing channel due to connection close
INFO:asyncssh:[conn=0, chan=0] Channel closed: Timeout, your session not responding.

On Windows SSH to the same server and staying idle (as long as the server allows) does not disconnect. Here, I could see from debug lines that the server sends packet type 98 SSH_MSG_CHANNEL_REQUEST, and client sends back packet type 100 SSH_MSG_CHANNEL_FAILURE. I am not sure if this is what keeps the session alive.

PS: C:\ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

PS: C:\ssh <problematic_server_IP> -vvv
... snip until prompt
< debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype [e-mail] reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype [e-mail] reply 1
debug3: send packet: type 100
...

Another weird thing is that the python code works okay when connected to a linux server. That is, client sessions kept alive as long as the server allows. So there is something else I need to do to keep the session alive for the problematic server.

Do you have any pointer to help me?

[ronf]

Wow - OpenSSH 4.3 is REALLY old. It was released over 15 years ago (Feb 1, 2006).

I think you may be running into https://bugzilla.mindrot.org/show_bug.cgi?id=1307, which was fixed in OpenSSH back in version 4.9 in 2008, or perhaps the change to accept more packet types was an accidental side effect of another keepalive-related change (and I see a handful of those over the years).

That said, if sending a MSG_CHANNEL_FAILURE instead of MSG_CHANNEL_SUCCESS in AsyncSSH is enough to work around the problem, I’d be willing to make the change to _process_keepalive_at_openssh_dot_com_request to do that, since modern versions of OpenSSH seems to accept either. There’s no need to submit a pull request here – I can take care of this after I get some other changes checked in. I’m hoping it’s just a one-line change.

[ronf]

This change is now available in AsyncSSH 2.9.0.