UI hangs after running multi-process optimization w/Optuna, joblib

Problem Description

After running an optimization using the joblib package and Optuna, the Spyder UI becomes unresponsive for long periods while completing post-processing code. The un-responsive period is after all the parallel python processes are shut down, which I monitor via Process Explorer. Sometimes it hangs completely and I have to kill the process and restart.

What steps reproduce the problem?

I don’t yet have a simple repro–I wanted to raise the issue and see if there is any awareness of a problem on your side. I could not find anything in historical issues. But basically I’m running xgboost models on multiple threads via joblib and Optuna, Optuna is using a SQLite dB to store the data. That part works fine, everything runs as expected, the dB is updated, etc.

What is the expected output? What do you see instead?

If I configure the same code to run in the CLI, and do so, it completes nearly immediately after all the threads are done and closed.

Versions

Note: I have been seeing this for more than 1 year in various versions of Spyder.

Python 3.8.8 64-bit
Qt 5.12.10
PyQt5 5.12.3
Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  3.0.4 (OK)
cloudpickle >=0.5.0             :  1.4.1 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.8.0 (OK)
pylint >=1.0                    :  2.7.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.2 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
setuptools >=39.0.0             :  52.0.0 (OK)
sphinx >=0.6.6                  :  3.5.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.2 (OK)
textdistance >=4.2.0            :  4.2.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  0.10.3 (OK)
zmq >=17                        :  22.0.3 (OK)

# Optional:
cython >=0.21                   :  0.29.14 (OK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.19.5 (OK)
pandas >=1.1.1                  :  1.2.3 (OK)
scipy >=0.17.0                  :  1.6.1 (OK)
sympy >=0.7.3                   :  None (NOK)

[eafpres]

Today I stripped down my code to just the parallel optimization and removed all unused functions in that minimal version. I have not been able to reproduce the issue so far. I’ll keep looking into it.

[eafpres]

So does the prompt in the IPython console take a lot to come back after you run your code?

Yes, but the entire UI becomes un-responsive. But I will get you an example; I just need some time; my working code is way too big and complex.

FWIW, the same behavior happens if the UI is open but I run the parallel process from another shell. So the interaction is occurring in some common process TBD.