Threadpools & NetMiko

Hey ktbyers - excellent module BTW! I’m trying to take my automation to the next level and use thread pools to get through a bunch of devices quick. I want to log in, check if the particular device is preconfigured for a log discriminator, and then either skip, or make the config changes. The script works fine if I go with one thread, 2 or more everything starts getting wacky. I’ve tried playing with timers but it doesn’t seem to affect anything. Should it matter this is over the cisco_ios_telnet type. Below is the function I seem to be having an issue with.

def verify_logging(HOST):
 output1 = net_con.send_command('sh run | i logging')
 sleep(2)
 if 'logging buffered discriminator VTY 25000' in output1:
  print(HOST + ' Logging pre-configured')
   completed(HOST)
 else:
  print('#################### ' + HOST + ' did not have enlarged buffer enabled ####################')
  print('Setting log buffer to 25k on ' + HOST)
  global set_logging
  set_logging = ['logging discriminator VTY msg-body drops VTY_ACCESS', 'logging buffered discriminator VTY 25000']#set log buff to 25k
  output = net_con.send_config_set(set_logging)
  outputclear = net_con.send_command('clear logging')
  if 'confirm' in outputclear:
   outputclear1 = net_con.send_command('y')
  print(outputclear1)
  output1 = net_con.send_command('wr')
  completed(HOST)

def main_thread(HOST):
 net_con = ConnectHandler(device_type=platform, ip=HOST, username=usern, password=passwd) #tacacs login
 print ("Logging in to " + HOST + " via TACACS")
 verify_logging(HOST)

What happens is some devices get updated correctly, others partially, and the rest not at all. I ran the logging on netmiko and the output file seems to show the hostname a few times and a bunch of read/writes on the channel:

DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b’\n’ DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b’\n’ DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b’\n’ DEBUG:netmiko:read_channel: RTR1# DEBUG:netmiko:read_channel: DEBUG:netmiko:read_channel:

Not sure what this could be. I’m fairly new to python so maybe I’m missing something. Other devices it does something like this and seemingly doubles and triples up on config in the same device:

RTR2(config)#logging buffered discriminator VTY 25000 RTR2(config)#logging buffered discriminator VTY 25000 RTR2(config)#logging buffered discriminator VTY 25000 RTR2(config)#logging buffered discriminator VTY 25000 RTR2(config)#logging buffered discriminator VTY 25000 RTR2(config)# RTR2# DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b’\n’ DEBUG:netmiko:read_channel: DEBUG:netmiko:write_channel: b’\r\n’ DEBUG:netmiko:read_channel:

Any ideas?

[johnarnold]

One thing in your example that was really confusing is that you use main_thread as a function name, but in fact, you’re running that function in the child threads. Typically, people refer to “main thread” and “main()” function together, meaning the parent thread + function that launches all the child threads.

Also, you can create connection handler objects and pass them into functions on child threads, (e.g. so you can re-use a ssh session) but you better be really sure that you’re being threadsafe (only reading/writing the object from one thread at a time). This is really easy to screw up.

Also, don’t use global vars. That’s a code smell unless you really know what you’re doing with them.

[degroat-c]

@johnarnold Thanks for all the tips. I certainly got messy with the code here on this one. So far after the help here and some cleanup everything seems to be running stable and clean. Thanks again!