Controller sometimes cannot start because of old controller or connections

We occasionally run into situations where tests fail because the controller hasn’t been shut down cleanly, like the following:

======================================================================  
ERROR: testLinear5 (__main__.testLinearOVSUser)  
Ping test on a 5-switch topology  
----------------------------------------------------------------------  
Traceback (most recent call last):  
  File "mininet/test/test_nets.py", line 70, in testLinear5  
    mn = Mininet( LinearTopo( k=5 ), self.switchClass, Host, Controller, waitConnected=True )  
  File "build/bdist.linux-i686/egg/mininet/net.py", line 170, in __init__  
    self.build()  
  File "build/bdist.linux-i686/egg/mininet/net.py", line 417, in build  
    self.buildFromTopo( self.topo )  
  File "build/bdist.linux-i686/egg/mininet/net.py", line 387, in buildFromTopo  
    self.addController( 'c%d' % i, cls )  
  File "build/bdist.linux-i686/egg/mininet/net.py", line 260, in addController  
    controller_new = controller( name, **params )  
  File "build/bdist.linux-i686/egg/mininet/node.py", line 1262, in __init__  
    self.checkListening()  
  File "build/bdist.linux-i686/egg/mininet/node.py", line 1280, in checkListening  
    '\n'.join( clist ) )  
Exception: Please shut down the controller which is running on port 6633:  
Active Internet connections (servers and established)   
tcp        0      0 127.0.0.1:60297         127.0.0.1:6633          TIME_WAIT   -                  
tcp        0      0 127.0.0.1:60296         127.0.0.1:6633          TIME_WAIT   -                  
tcp        0      0 127.0.0.1:60298         127.0.0.1:6633          TIME_WAIT   -                  
tcp        0      0 127.0.0.1:60299         127.0.0.1:6633          TIME_WAIT   -                  
tcp        0      0 127.0.0.1:60300         127.0.0.1:6633          TIME_WAIT   -                  

----------------------------------------------------------------------  
Ran 12 tests in -25078.580s  

Casting aside the clock problem (we can’t reliably set the time in Ubuntu without it getting messed up), it seems that there is a race condition where our switches and/or controller are supposed to have shut down but starting up a new controller fails because the old controller still seems to be running and/or it hasn’t released its port.

[maikarpey]

Execute the following command to kill it: sudo fuser -k 6633/tcp

[bidhovbizar]

Run netstat -ltnp | grep ‘6653’ -(1)

which will give you the process id which is using that port. If you find that PID as 1234 then run

kill -9 1234. -(2)

Recheck with (1) to see if the process is actually dead or did somethng else bring it back up.