Small device input buffers and send_config_set failures

We’ve recently encountered issues where we send a long list of commands to send_config_set and a remote device appears to fill up the input buffers on the device, causing it to not properly execute commands.

This appears to be an artifact of how send_config_set operates, by sending the entire list of commands to the device, then grabbing the entire output afterwards.

I believe in instances such as this, we could batch the commands, grab output, then proceed to the next batch of commands until the entire list of commands has been executed.

My thought is something like this:

def send_config_set(
    self,
    config_commands=None,
    exit_config_mode=True,
    delay_factor=1,
    max_loops=150,
    strip_prompt=False,
    strip_command=False,
    config_mode_command=None,
    batch_commands=False,
    batch_length=10,
):
    import math
    """
    Send configuration commands down the SSH channel.
    config_commands is an iterable containing all of the configuration commands.
    The commands will be executed one after the other.
    Automatically exits/enters configuration mode.
    :param config_commands: Multiple configuration commands to be sent to the device
    :type config_commands: list or string
    :param exit_config_mode: Determines whether or not to exit config mode after complete
    :type exit_config_mode: bool
    :param delay_factor: Factor to adjust delays
    :type delay_factor: int
    :param max_loops: Controls wait time in conjunction with delay_factor (default: 150)
    :type max_loops: int
    :param strip_prompt: Determines whether or not to strip the prompt
    :type strip_prompt: bool
    :param strip_command: Determines whether or not to strip the command
    :type strip_command: bool
    :param config_mode_command: The command to enter into config mode
    :type config_mode_command: str
    :param batch_commands: Break the list of commands into batches for execution
    :type batch_commands: bool
    :param batch_length: Maximum length of each command batch
    :type batch_length: int
    """
    delay_factor = self.select_delay_factor(delay_factor)
    if config_commands is None:
        return ""
    elif isinstance(config_commands, string_types):
        config_commands = (config_commands,)

    if not hasattr(config_commands, "__iter__"):
        raise ValueError("Invalid argument passed into send_config_set")

    batch_length = batch_length if batch_commands else len(config_commands)
    batches = math.ceil(len(config_commands) / batch_length)
    start_batch = 0
    end_batch = batch_length
    fast_cli = False if batches > 1

    # Send config commands
    cfg_mode_args = (config_mode_command,) if config_mode_command else tuple()
    output = self.config_mode(*cfg_mode_args)
    for batch in range(batches):
        for cmd in config_commands[start_batch:end_batch]:
            self.write_channel(self.normalize_cmd(cmd))
            if self.fast_cli:
                pass
            else:
                time.sleep(delay_factor * 0.05)
        start_batch = end_batch
        end_batch += batch_length

        # Gather output
        output += self._read_channel_timing(
            delay_factor=delay_factor, max_loops=max_loops
        )

    if exit_config_mode:
        output += self.exit_config_mode()
    output = self._sanitize_output(output)
    log.debug("{}".format(output))
    return output

I plan on enhancing our in house integration tests to attempt to replicate this issue, so that I can try this out to see if it works properly. If it works and you don’t have a better solution, I’ll look at creating a PR.

[aedwardstx]

I’ll be monkey patching a more mature version of what I shared into our code base. Once done, I’ll submit a PR so you can make a final decision.

Addressing your points:

1. We don’t need to know the buffer sizes, the chunk_size can be set to 10. This ensures that there are only 10 unacknowledged commands pending at a time.
2. I’ll be working out the kinks for IOS, NXOS, IOS-XR, and EOS which I imagine is the bulk of your user base. I’ll be using this code to operate against 10,000+ devices. The entry point for this method could be separate send_config_set_with_arbiter as to not muddy the waters.
3. I think every network engineer has had the experience of pasting a large config too quickly and application of that config getting garbled or commands dropped. The lack of something intelligently pacing config commands sent impacted the application of an ACL on several devices in our environment. We frequently see cases where ACL entries are slow to apply since the device has to program TCAM on the fly.
4. a. Appealing idea but send_config_set should still be a reliable method. b. The Arbiter method removes the need to constantly tune timers. Instead, it sends config as fast as possible and slows down automatically when it needs to. c. Netmiko send_config_set should be reliable for one shot configuration changes regardless of the size of the change set.

[ktbyers]

Adding same comment here as I put on the PR #1085

---

I know this is not exactly what you were looking for, but we fixed this problem by essentially going to a command-buffer of one i.e. Netmiko won’t write the next command in send_config_set until the last one has been properly echoed/handled by the device.

That fix is in the Netmiko develop branch and there is a modification to the behavior here:

#1447

I need to make some slight changes to that PR1447 as I have hard-coded a terminating char ‘#’ in there, but I will update that shortly.