SSH is_alive() causes slow Cisco IOS device to get out of sync with netmiko

Because the null character sent by is_alive() on SSH sessions causes the IOS device to echo a new prompt (as noted in https://github.com/ktbyers/netmiko/issues/568#issuecomment-329571918), if the device is sufficiently slow in responding, the device can get “out of sync” to where, in looking for the prompt, it hits one of these extra echoed prompts and stops there instead of reading to the latest (expected) prompt.

Here’s an excerpt of a session illustrating this problem:

[28/Mar/2019 20:03:26] DEBUG [netmiko:988] Command: terminal length 0

[28/Mar/2019 20:03:26] DEBUG [netmiko:385] write_channel: b'terminal length 0\n'
[28/Mar/2019 20:03:26] DEBUG [netmiko:506] Pattern is: csr1000v
[28/Mar/2019 20:03:26] DEBUG [netmiko:523] _read_channel_expect read_data: t
[28/Mar/2019 20:03:26] DEBUG [netmiko:523] _read_channel_expect read_data: erminal le
[28/Mar/2019 20:03:26] DEBUG [netmiko:523] _read_channel_expect read_data: ng
[28/Mar/2019 20:03:26] DEBUG [netmiko:523] _read_channel_expect read_data: th
[28/Mar/2019 20:03:27] DEBUG [netmiko:523] _read_channel_expect read_data:  0
csr1000v#
[28/Mar/2019 20:03:27] DEBUG [netmiko:535] Pattern found: csr1000v terminal length 0
csr1000v#
[28/Mar/2019 20:03:27] DEBUG [netmiko:993] terminal length 0
csr1000v#
[28/Mar/2019 20:03:27] DEBUG [netmiko:994] Exiting disable_paging
[28/Mar/2019 20:03:27] DEBUG [netmiko:385] write_channel: b'terminal width 511\n'
[28/Mar/2019 20:03:27] DEBUG [netmiko:506] Pattern is: csr1000v
[28/Mar/2019 20:03:27] DEBUG [netmiko:523] _read_channel_expect read_data: t
[28/Mar/2019 20:03:27] DEBUG [netmiko:523] _read_channel_expect read_data: erminal
[28/Mar/2019 20:03:28] DEBUG [netmiko:523] _read_channel_expect read_data:  widt
[28/Mar/2019 20:03:28] DEBUG [netmiko:523] _read_channel_expect read_data: h 5
[28/Mar/2019 20:03:29] DEBUG [netmiko:523] _read_channel_expect read_data: 1
[28/Mar/2019 20:03:29] DEBUG [netmiko:523] _read_channel_expect read_data: 1
csr1000v#
[28/Mar/2019 20:03:29] DEBUG [netmiko:535] Pattern found: csr1000v terminal width 511
csr1000v#
[28/Mar/2019 20:03:30] DEBUG [netmiko:463] read_channel: 
[28/Mar/2019 20:03:30] DEBUG [netmiko:434] Sending the NULL byte
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'\x00'
[28/Mar/2019 20:03:30] DEBUG [netmiko:434] Sending the NULL byte
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'\x00'
[28/Mar/2019 20:03:30] DEBUG [netmiko:434] Sending the NULL byte
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'\x00'

Here, after establishing the session (above), we call send_config_set(), but observe what happens:

[28/Mar/2019 20:03:30] DEBUG [netmiko:1548] ---> send_config_set()
[28/Mar/2019 20:03:30] DEBUG [netmiko:1463] ---> config_mode()
[28/Mar/2019 20:03:30] DEBUG [netmiko:1444] ---> check_config_mode()
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'\n'
[28/Mar/2019 20:03:30] DEBUG [netmiko:506] Pattern is: #
[28/Mar/2019 20:03:30] DEBUG [netmiko:523] _read_channel_expect read_data: ^@
csr1000v#
[28/Mar/2019 20:03:30] DEBUG [netmiko:535] Pattern found: # ^@
csr1000v#
[28/Mar/2019 20:03:30] DEBUG [netmiko:1451] <--- check_config_mode()
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'config term\n'
[28/Mar/2019 20:03:30] DEBUG [netmiko:506] Pattern is: csr1000v
[28/Mar/2019 20:03:30] DEBUG [netmiko:523] _read_channel_expect read_data: ^@
[28/Mar/2019 20:03:30] DEBUG [netmiko:523] _read_channel_expect read_data: 
csr1000v#^@
csr1000v#
[28/Mar/2019 20:03:30] DEBUG [netmiko:535] Pattern found: csr1000v ^@
csr1000v#^@
csr1000v#

Note that we just sent config term\n but the prompt that we just read back was one induced by is_alive() (see the ^@ character) - the output from the device hasn’t yet reached the point where it echoed config term followed by the actual latest prompt.

Continuing…

[28/Mar/2019 20:03:30] DEBUG [netmiko:1444] ---> check_config_mode()
[28/Mar/2019 20:03:30] DEBUG [netmiko:385] write_channel: b'\n'
[28/Mar/2019 20:03:30] DEBUG [netmiko:506] Pattern is: #
[28/Mar/2019 20:03:31] DEBUG [netmiko:523] _read_channel_expect read_data: 

[28/Mar/2019 20:03:31] DEBUG [netmiko:523] _read_channel_expect read_data: csr1000v#confi
[28/Mar/2019 20:03:31] DEBUG [netmiko:535] Pattern found: # 
csr1000v#confi
[28/Mar/2019 20:03:31] DEBUG [netmiko:1451] <--- check_config_mode()

At this point, the check_config_mode() call after sending “config term” fails here, because due to the out-of-sync situation, it has found a non-config-mode prompt and hence thinks the system was unable to enter config mode (when in fact the output that’s being read from the device simply hasn’t reached that point yet).

[glennmatthews]

Got it. I’ll rip out the is_alive check from our code and we can address any issues as they arise. Thanks for the help!

[ktbyers]

@glennmatthews I am open to trying to figure out a better solution. I just know up until now is_alive() has been problematic.