Process completes but does not exit (leaving stranded workers)

/usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1

plaso-1.3.1.post20151021

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored
Exception TypeError: "'NoneType' object is not callable" in <Finalize object, dead> ignored\


analyst   4035  3335  0 Oct29 pts/19   00:00:57 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4045  4035  0 Oct29 pts/19   00:01:02 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4048  4035  0 Oct29 pts/19   00:00:39 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4051  4035  0 Oct29 pts/19   00:00:58 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4054  4035  0 Oct29 pts/19   00:00:41 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4060  4035  0 Oct29 pts/19   00:00:44 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4063  4035  0 Oct29 pts/19   00:00:40 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4066  4035  0 Oct29 pts/19   00:00:37 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4068  4035  0 Oct29 pts/19   00:00:40 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4070  4035  0 Oct29 pts/19   00:11:09 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4073  4035  0 Oct29 pts/19   00:00:37 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4076  4035  0 Oct29 pts/19   00:00:37 /usr/bin/python /usr/local/bin/log2timeline.py -p --vss-stores 1 srt.dump /media/usb1
analyst   4083  4035  1 Oct29 pts/19   00:14:40 [log2timeline.py] <defunct>

[butljon]

thanks again for the tip … and there I found it! wrong in my program was that one needs to get all the stuff off the queue before joining the threads (otherwise threads may not be able to terminate): http://stackoverflow.com/questions/31170788/python-multiprocessing-threads-dont-close-when-using-queue

This is most likely because of this documented quirk of multiprocessing.Queue:

Bear in mind that a process that has put items in a queue will wait before terminating until all the
buffered items are fed by the “feeder” thread to the underlying pipe. (The child process can call the
cancel_join_thread() method of the queue to avoid this behaviour.)

This means that whenever you use a queue you need to make sure that all items which have been
put on the queue will eventually be removed before the process is joined. Otherwise you cannot be
sure that processes which have put items on the queue will terminate. Remember also that
non-daemonic processes will be joined automatically.

Basically, you need to make sure you get() all the items from a Queue to guarantee that all the
processes which put something into that Queue will be able to exit.

[joachimmetz]

Closing this issue, behavior of multi processing has changed due to: https://github.com/log2timeline/plaso/issues/338