Implement task timeouts

Timeouts are an important feature for task runs. There are some difficulties in implementing this correctly though. Here is a quick and easy implementation inside of a task runner:

import signal

def timeout_handler(signum, frame):                                                 
    raise TimeoutError("Execution timed out.")                                      
                                                                                                                                                               
@contextmanager                                                                     
def timeout(seconds=None):                                                          
    if seconds:                                                                     
        try:                                                                        
            signal.signal(signal.SIGALRM, timeout_handler)                          
            signal.alarm(round(seconds))                                            
            yield                                                                   
        finally:                                                                    
            signal.alarm(0)                                                         
    else:                                                                           
        yield  

...
with timeout(self.timeout):                                      
    state = self.get_run_state(state=state, inputs=task_inputs)

this works great for the LocalExecutor and SynchronousExecutor, but fails for DaskExecutor because signal cannot be called from outside the main thread.

Other possibilities:

• using async to submit task.run with a timeout, but this would likely break our python 3.4 compatibility
• sending task.run to a new process using multiprocessing.Process; this is probably the route we will eventually go but note: prints to standard out (which we rely upon for lots of tutorials) will break. Moreover, retrieving the return value from the task.run method is non-trivial and requires additional communication overhead

[jlowin]

It’s very nice to be able to use (for example) LocalExecutor for debugging, which also implies working in the same process. That’s an argument for two different implementations (or possibly adding the multiprocessing solution on BaseExecutor and then overriding just for LocalExecutor.

[cicdw]

Sorry somewhere in the mix I got ahead of myself; I don’t think there’s actually a solution in the local case when DaskExecutor(processes=True). In that case, signal doesn’t work because it isn’t run in the main thread, and the multiprocessing spawning doesn’t work because you can’t spawn a process from within a daemonic subprocess. If the dask cluster was actually distributed (and not local), and the workers had the no-nanny flag set, then the multiprocessing solution should work.

I might go ahead and submit a PR that works for all executors other than that one and we can refactor as necessary.