HPC cluster workers preload file not found

I’m launching multiple (in this case 40, but it doesn’t matter) DASK workers at the same time on a HPC using a job scheduler. They start running almost at the same time and are preloading the same dask_preload.py file. Some (approximately half of them) workers die immediately with the following error, the others start up correctly:

Traceback (most recent call last):
  File "/python_virtualenv/bin/dask-worker", line 11, in <module>
    sys.exit(go())
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/cli/dask_worker.py", line 252, in go
    main()
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 696, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 621, in make_context
    self.parse_args(ctx, args)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 880, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 1404, in handle_parse_result
    self.callback, ctx, self, value)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 78, in invoke_param_callback
    return callback(ctx, param, value)
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/preloading.py", line 32, in validate_preload_argv
    preload_modules = _import_modules(ctx.params.get("preload"))
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/preloading.py", line 92, in _import_modules
    module = import_file(name)[0]
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/utils.py", line 1003, in import_file
    os.remove(cache_file)
OSError: [Errno 2] No such file or directory: 'dask_preload.pyc'

I can schedule workers to start 1 at a time. But first I want to know if it is possible to fix this within the scheduler code itself?

Btw. If I start 1 worker, it is able to load the dask_preload.py file.

[mrocklin]

Yes, what @pitrou proposes is an easy fix. I have a fix locally that I’ll push up in a bit.

On Tue, Apr 17, 2018 at 2:08 AM, Wouter van Roosmalen < [e-mail]> wrote:

Is this something to change in a next release? Or a custom change specific to this use case?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/1915#issuecomment-381857309, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszBv1_dc_eQuI2YmpXk1QmO9aa9Ifks5tpYbwgaJpZM4TRkBV .

[pitrou]

This looks like a benign case of race condition in https://github.com/dask/distributed/blob/master/distributed/utils.py#L1000-L1003.

Just replace:

if os.path.exists(cache_file):
    os.remove(cache_file)

with:

try:
    os.remove(cache_file)
except OSError:
    pass  # ignore "file not found"

(you can refine the exception check by querying errno)