Compiling skimage (scikit-image) | Nuitka 0.6.6rc7 | dask.config FileNotFoundError dask.yaml

Versions

0.6.6rc7 Python: 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] Executable: C:\Users\GREEJK\AppData\Local\Continuum\anaconda3\python.exe OS: Windows Arch: x86_64

Background

I apologize if the following background is too detailed. I wanted to provide sufficient information on my setup leading to the current problem, and some of it is included simply to leave hints for others like myself trying to get started with Nuitka. For TL;DR, skip to The Problem.

I am using Anaconda and I installed Nuitka 0.6.5 via conda install -c conda-forge nuitka There were some problems with testing, but after using conda install --force-reinstall -c conda-forge nuitka it was good.

Pretty much all the problems I’ve run into are with skimage (scikit-image 0.15.0). skimage is big and it seems to want to compile everything that belongs to it. (I don’t know if there is a way to get around that.) I have a shortened test case, test_skimage.py:

# -*- coding: utf-8 -*-
import skimage.io
image = skimage.io.imread("C:/Users/Username/Desktop/folder/image.ico")
print(image)

I tried compiling with python -m nuitka --standalone --mingw64 --plugin-enable=numpy --plugin-enable=qt-plugins --plugin-enable=pylint-warnings --experimental=use_pefile Desktop/folder/test_skimage.py. This run-time error occurs:

Traceback (most recent call last):
  File "C:\Users\GREEJK\test_skimage.dist\test_skimage.py", line 5, in <module>
  File "C:\Users\GREEJK\test_skimage.dist\skimage\io\_io.py", line 48, in imread
  File "C:\Users\GREEJK\test_skimage.dist\skimage\io\manage_plugins.py", line 197, in call_plugin
RuntimeError: No suitable plugin registered for imread.

You may load I/O plugins with the `skimage.io.use_plugin` command.  A list of all available plugins are shown in the `skimage.io` docstring.

Side note: In attempt to figure out the runtime error, I edited test_skimage.py by puting skimage.io.use_plugin('imread', 'imread') before the call to skimage.io.imread. It gives this error at runtime:

Traceback (most recent call last):
  File "C:\Users\GREEJK\test_skimage.dist\test_skimage.py", line 5, in <module>
  File "C:\Users\GREEJK\test_skimage.dist\skimage\io\manage_plugins.py", line 245, in use_plugin
KeyError: 'imread'

The same thing happens if I use skimage.io.use_plugin('pil', 'imread'). This error does not occur when simply running the script. --only in the compiled program.

Seeing that perhaps one or more problems with skimage have been fixed in the development version of Nuitka, I installed 0.6.6rc7. I used pip install -U "https://github.com/Nuitka/Nuitka/archive/develop.zip" to do so.

With python -m nuitka --standalone --mingw64 --plugin-enable=numpy --plugin-enable=qt-plugins --plugin-enable=pylint-warnings --include-plugin-directory=AppData\Local\Continuum\anaconda3\Lib\site-packages\skimage\io\_plugins --experimental=use_pefile Desktop/folder/test_skimage.py I get the following output:

Nuitka:INFO:Injecting plug-in based pre load code for module 'PyQt5':
Nuitka:INFO:    Adding dist folder to 'PATH' environment variable.
Nuitka:INFO:Injecting plug-in based post load code for module 'PyQt5.QtCore':
Nuitka:INFO:    Setting Qt library path to distribution folder. Need to avoid loading target
Nuitka:INFO:    system Qt plug-ins, which may be from another Qt version.
Traceback (most recent call last):
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\__main__.py", line 184, in <module>
    main()
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\__main__.py", line 177, in main
    MainControl.main()
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\MainControl.py", line 788, in main
    executePostProcessing(getResultFullpath(main_module))
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\PostProcessing.py", line 65, in executePostProcessing
    lang_id=0,
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\utils\WindowsResources.py", line 247, in addResourceToFile
    _closeFileWindowsResources(update_handle)
  File "C:\Users\Username\AppData\Local\Continuum\anaconda3\lib\site-packages\nuitka\utils\WindowsResources.py", line 165, in _closeFileWindowsResources
    raise ctypes.WinError()
OSError: [WinError 110] The system cannot open the device or file specified.

Another side note: Including the parameter --include-plugin-directory=AppData\Local\Continuum\anaconda3\Lib\site-packages\skimage\io\_plugins gets rid of all the “Unresolved ‘__import__’ call” warnings. Maybe that’s an important detail, though it seems mostly aesthetic.

Following a work-around from https://github.com/Nuitka/Nuitka/issues/468, I added the following code below line 253 of anaconda3\pkgs\nuitka-0.6.5-py_0\site-packages\nuitka\utils\WindowsResources.py AND below line 242 of anaconda3\Lib\site-packages\nuitka\utils\WindowsResources.py (I’m pretty sure the latter is the one that matters. It holds the version of Nuitka I’m using.):

import os
target_filename = os.path.abspath(target_filename)

The Problem

It now compiles without errors. When I execute the program, I get this error:

Traceback (most recent call last):
  File "C:\Users\Username\test_skimage.dist\test_skimage.py", line 3, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\Username\test_skimage.dist\skimage\__init__.py", line 127, in <module skimage>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\Username\test_skimage.dist\skimage\util\__init__.py", line 6, in <module skimage.util>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\Username\test_skimage.dist\skimage\util\apply_parallel.py", line 8, in <module skimage.util.apply_parallel>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\Username\test_skimage.dist\dask\__init__.py", line 1, in <module dask>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "C:\Users\Username\test_skimage.dist\dask\config.py", line 532, in <module dask.config>
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Username\\test_skimage.dist\\dask\\dask.yaml'

It’s not one I understand yet.

[JorjMcKie]

Since a long time, scipy is a part of the numpy plugin. So you should use --enable-plugin=numpy=scipy. Also to you the recommendation to use hinted compilation, which would remove a lot of headache in this respect.

[JorjMcKie]

To my surprise, only on a conda installation the package dask is part of the dependencies of skimage! And dask has that yaml configuration file, which - as a non-Python file - must be copied into the .dist folder. A no-brainer once this fact is known. Now checking if other specialties come with dask before I upload new plugin versions.