client.submit not working with client.upload_file after first time

Below are my local and cloud Dask version.

Dask version: 2.14 Distributed version: 2.14

We have a production Dask cluster in GKE, and shared among our engineers. Now, we want to create a python script for boilerplate purpose, simply name, boilerplate.py,

def inc(x):
    if x > 10:
        raise Exception('x > 10')
    return x + 1

And our client script,

from dask.distributed import Client
import boilerplate as bp
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')

client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error

Now we fixed boilerplate.py,

def inc(x):
    if x > 20:
        raise Exception('x > 20')
    return x + 1

And rerun client script to reupload fixed boilerplate.py,

from dask.distributed import Client
Exception: x > 10
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')

client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error

It is still same exception,

Exception: x > 10

The problem here, we cannot restart our Dask cluster, it might disturb other important tasks. It might seems got some caching layer here, this is only happened if we use file_upload + submit, if we use file_upload + run, it use latest update boilerplate.py.

[TomAugspurger]

Let us know if you’re able to do more debugging.

Given the complexities here, I don’t really see how Dask can support re-uploading modules with upload_file. I’d recommend investigating alternative ways of doing this.

We can reopen if you find a specific issue or a solution that works well generally.

[huseinzol05]

@TomAugspurger