Stopping a submitted function in Jupyter Notebook results in <FutureState: error> and a needed restart

The client enters a <FutureState: error> state when a job is submitted to the client and the kernel is stopped.

This is a problem because once in that state, no other jobs can be completed (client.gather(result) blocks but does not complete for any submitted future jobs).

The only way to progress is to restart the Jupyter notebook kernel.

Instructions to reproduce, paste this code into a Jupyter Notebook cell:

import time
from dask.distributed import Client

client = Client()

def add(a,b):
    time.sleep(60)
    return a+b

def square(a):
    return a**2

Then run the below code and stop the kernel prior to the function completing (<60s).

result = client.submit(add, 1, 4)
client.gather(result)

Checking the client futures, we can see an error FutureState:

>>> client.futures
{'add-962bd7f68184e275518de72ad6b32ed9': <FutureState: error>}

Attempting to execute other functions thereafter results in the client.gather function blocking any further execution:

result2 = client.submit(square, 5)
client.gather(result2)

The only way to revert back to the expected behaviour is to restart the Jupyter notebook kernel (which is not ideal if a lot of data has been loaded from disk)

[mrocklin]

By the way @gregsifr thank you for producing such a clean example of this issue

[GenevieveBuckley]

It looks like the patch to CPython has been applied and released, so I’ll close this issue. Thanks @gregsifr