Compiled modules of non-default interpreters fail to import on Windows

Problem Description

Spyder v3.3.3 not loading numpy from different conda environment.

All modules installed using default channels (.condarc). I have read a recent support ticket and have followed the modular approach (detailed below). This doesn’t work.

What steps reproduce the problem?

I have a fresh install of Miniconda3. I have created a env called spyder3 and installed spyder. I have then created another env called test and installed the spyder kernels and numpy.

1. Download and install miniconda3
2. Update conda: conda update conda -y
3. Created an env called spyder3: conda create --name spyder3 -y
4. Activate spyder3 env: conda activate spyder3
5. Install spyder: conda install spyder
6. Deactivate spyder3 env: conda deactivate
7. Create a new env called test2: conda create --name test2 -y
8. Activate test2: conda activate test2
9. Install spyder kernels: conda install spyder-kernels=0.* -y
10. Install numpy: conda install numpy
11. Deactivate test2 env: conda deactivate
12. Activate spyder3 env: conda activate spyder3
13. Run spyder: spyder
14. Change python interpreter: Tools>Preferences>Python interpreter>…pointed to …Miniconda3\envs\test2\python.exe.
15. Open new console
16. Import numpy in console

What is the expected output? What do you see instead?

Expecting numpy to load without error like other modules (example: xlsxwriter).

Paste Traceback/Error Below (if applicable)spyder

Traceback (most recent call last):

File “<ipython-input-1-d4cdadb62aa7>”, line 1, in <module> import numpy

File “C:\Users\dunca\Miniconda3\envs\test2\lib\site-packages\numpy_init_.py”, line 140, in <module> from . import _distributor_init

File “C:\Users\dunca\Miniconda3\envs\test2\lib\site-packages\numpy_distributor_init.py”, line 34, in <module> from . import _mklinit

ImportError: DLL load failed: The specified module could not be found.

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 3.3.3
* Python version: 3.7.3 64bit
* Qt version: 5.9.6
* PyQt version: 5.9.2
* Operating System name/version: Windows 10

### Dependencies
IPython >=4.0     :  7.4.0 (OK)
cython >=0.21     :  None (NOK)
jedi >=0.9.0      :  0.13.3 (OK)
matplotlib >=2.0.0:  None (NOK)
nbconvert >=4.0   :  5.4.1 (OK)
numpy >=1.7       :  None (NOK)
pandas >=0.13.1   :  None (NOK)
pycodestyle >=2.3 :  2.5.0 (OK)
pyflakes >=0.6.0  :  2.1.1 (OK)
pygments >=2.0    :  2.3.1 (OK)
pylint >=0.25     :  2.3.1 (OK)
qtconsole >=4.2.0 :  4.4.3 (OK)
rope >=0.9.4      :  0.12.0 (OK)
sphinx >=0.6.6    :  1.8.5 (OK)
sympy >=0.7.3     :  None (NOK)

[ccordoba12]

Thanks for reporting. This was also found by one of our developers (@CAM-Gerlach) a few weeks ago. We’ll fix it in our next version (3.3.5)

[CAM-Gerlach]

Thanks for your response, @mingwandroid . To be clear, I didn’t mean to call you over here to accuse you or Anaconda of causing this issue without robust evidence, and I don’t recall previously making that claim @ccordoba12 mentioned above (though its not impossible I could have forgotten). I only did so due to knowing your deep expertise in this area and the fact you’d been summoned to take a look at DLL loading issues previously. I just wanted to preemptively apologize if you garnered that impression from any of the posts here.

---

I did extensive testing on Windows 8.1 x64 Miniconda x64, with Spyder running under both Python 3.7.2-h8c8aaf0_2 with the DLL search order enabled by default, and the latest 3.7.3-h8c8aaf0_1 with it disabled, and replicated in two different interpreter environments (in addition to Spyder’s own), a wide variety of Python builds/versions and in some cases multiple repeat trials of each. In summary, the results support the DLL modification I was referring to being the main culprit when enabled (i.e. on 3.7.2-h8c8aaf0_2 and 3.6.8-h9f7ef89_1 by default, both of which I had tested on previously), and secondarily Python 3.7 not adding the interpreter env to the PATH while Python 3.6 did, which spyder-ide/spyder-kernels#109 fixes so that this error does not occur except on those two Python versions, or with CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 system-wide.

Full details:

Without spyder-ide/spyder-kernels#109 applied (i.e. running Spyder-Kernels 1.3.2), I tested the following Python versions for this problem [with various affected packages; all versions given are kernel interpreter versions, not the version Spyder is running on]:

3.7.3-h8c8aaf0_1  ERROR
3.7.3-h8c8aaf0_0  ERROR
3.7.2-h8c8aaf0_10 ERROR
3.7.2-h8c8aaf0_2  ERROR *DLL Change on by default*
3.7.2-h8c8aaf0_0  ERROR
3.7.1-h8c8aaf0_6  ERROR
3.7.0-hea74fb7_0  ERROR

3.6.8-h9f7ef89_7  GOOD
3.6.8-h9f7ef89_1  ERROR *DLL Change on by default*
3.6.8-h9f7ef89_0  GOOD
3.6.7-h9f7ef89_2  GOOD
3.6.7-h33f27b4_1  GOOD

However, I noticed that os.environ["PATH"] included C:\\Miniconda3\\envs\\INTERPRETER_ENV\\Library\\bin on Python 3.6. while it only had references to Spyder’s env on Python 3.7 (there were no other differences than this one entry, and this was consistent between DLL search change on by default and all other builds). Manually adding the former to the latter got me the exact same results as with the spyder-kernels PR applied (see below); i.e. changing it manually to include that path did not work with the DLL search path change enabled but did work otherwise.

---

With that PR applied (adding the relevant PATH entries) iIt doesn’t appear to be an issue with the logic adding the entries to the path, since they are still added just the same on all the builds, good or error (and is_anaconda(), is_conda = osp.exists(osp.join(sys.prefix, 'conda-meta')), returns True in either case):

3.7.3-h8c8aaf0_1  GOOD
3.7.3-h8c8aaf0_0  GOOD
3.7.2-h8c8aaf0_10 GOOD
3.7.2-h8c8aaf0_2  ERROR *DLL Change on by default*
3.7.2-h8c8aaf0_0  GOOD
3.7.1-h8c8aaf0_6  GOOD

3.6.8-h9f7ef89_7  GOOD
3.6.8-h9f7ef89_1  ERROR *DLL Change on by default*
3.6.8-h9f7ef89_0  GOOD
3.6.7-h9f7ef89_2  GOOD
3.6.6-hea74fb7_0  GOOD
3.6.5-h0c2934d_0  GOOD

---

Can I ask that you be specific about which change you mean here?

I was referring to this, which does indeed to be the primary source of the problem that doesn’t appear easily solvable on our side.

the DLL changes (by which I mean the dropping of the automatic addition of of Library/bin to PATH) were done more than a year ago now.

I’m referring to the changed prompted in part by spyder-ide/spyder#7357 , which dropped in mid-February (less than three months ago), and the exact versions of Python mentioned that experiences the aberrant behavior are the ones you refer to in which it was enabled by default. Therefore, it seems having that enabled is the root of the issue here.

I believe that what changed was that conda 4.6.0 to 4.6.13 inclusive had a bug to do with activation mishandling PATH on Windows.

This problem has occurred consistently under conda 4.5.12 and 4.5.13. I don’t believe there’s any direct interaction with the conda executable involved here currently, as Spyder does not call it to activate the kernel environment (if it did, it seems doubtful we would be having this problem).

Forgive me if I misunderstand here, but I think Spyder wants to be stacking conda environments to achieve this single Spyder env, multiple kernels in multiple other envs?

Ideally, it would only have the PATH entries for the kernel’s env, since it should only be able to access packages and binaries in the current running environment as would be the case if it had been activated normally from the Anaconda prompt. Otherwise, theoretically at least, packages could work (or otherwise behave differently) if they have access to DLLs in Spyder’s env that are not actually present in the “isolated” kernel env. Doesn’t seem like a very big problem, but it also makes the PATH longer than it needs to be and there’s always some chance of a bug lurking there.