ModuleNotFoundError: No module named 'dask_xgboost' during worker pickle.loads()

Running the Helm Chart latest version, with:

    - name: EXTRA_PIP_PACKAGES
      value: s3fs dask-ml dask-xgboost --upgrade

k8s Dask worker pod logs confirm these were all installed on the workers. I also ran a job pip show ... for these to check.

Despite that, when running a notebook with dask_xgboost, I get this in the worker:

/opt/conda/lib/python3.7/site-packages/tornado/gen.py in wrapper()
    207                 # performance penalty for the synchronous case.
    208                 try:
--> 209                     yielded = next(result)
    210                 except (StopIteration, Return) as e:
    211                     future_set_result_unless_cancelled(

/opt/conda/lib/python3.7/site-packages/distributed/worker.py in run()
   2923 @gen.coroutine
   2924 def run(server, comm, function, args=(), kwargs={}, is_coro=None, wait=True):
-> 2925     function = pickle.loads(function)
   2926     if is_coro is None:
   2927         is_coro = iscoroutinefunction(function)

/opt/conda/lib/python3.7/site-packages/distributed/protocol/pickle.py in loads()
     57 def loads(x):
     58     try:
---> 59         return pickle.loads(x)
     60     except Exception:
     61         logger.info("Failed to deserialize %s", x[:10000], exc_info=True)

ModuleNotFoundError: No module named 'dask_xgboost'

I’m more than a little stumped … all I can guess is that somehow this module is imported twice in incompatible ways … maybe the dask_ml import into dask_ml.xgboost?

[mrocklin]

Ah, it usually doesn’t need those libraries but in the case of dask-xgboost, the scheduler is used to host the XGBoost master node

On Sun, Apr 14, 2019 at 6:16 PM Brett Randall [e-mail] wrote:

Ok, I managed to get past this problem, but I’m not sure what the fix implies. I had dask_xgboost installing as EXTRA_PIP_PACKAGES on both client dask-notebook and on Dask Workers, but not on the Dask Scheduler. Installing dask_xgboost on the Scheduler pod/container as well resolves this problem.

Should that be required? I’m not across everything the Scheduler does, but is it involved in:

• pickling/unpickling, as opposed to that being done in the Worker.
• Masquerading as an additional Worker sometimes.

If it is already known that the Scheduler should always have the complete/same set of modules as Workers (and Client), then this is a close. Thanks for the advice - it made me check what pods these were not installed in.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/dask-kubernetes/issues/134#issuecomment-483067039, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszE4hFOslKlLzfXp3jSv2vsqLri_hks5vg7bSgaJpZM4cup-g .

[quasiben]

@stromal I’d recommend opening a new issue with a minimal reproducible example if you are having problems. This issue is over a year old