Missing support for importing things from ".egg" files in sys.path

• Nuitka version, full Python version and Platform (Windows, OSX, Linux …) Nuitka 0.5.30 Python 2.7.12 pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7) Platform Ubuntu Linux

I installed Nuitka by downloading source tar and doing python setup.py install I’m trying to convert py-webrtcvad module to C

Steps to test the issue

git clone https://github.com/wiseman/py-webrtcvad
cd py-webrtcvad/
python example.py

If you get any import errors, do pip install webrtcvad

You should see the output

Usage: example.py <aggressiveness> <path to wav file>

Now, I try to convert example.py to C using following command

python -m nuitka --show-progress --show-modules --standalone --portable --recurse-all --python-version=2.7 example.py >& nuikta.build.log &

The build log is kept here

Main warnings from log are

Nuitka:WARNING:/home/user/code/python/webrtc_vad/testing/py-webrtcvad/webrtcvad.py:3: Cannot find '_webrtcvad' as relative or absolute import.
Nuitka:WARNING:/usr/local/lib/python2.7/dist-packages/pkg_resources/_vendor/appdirs.py:484: Cannot find 'com' in package 'pkg_resources._vendor' as relative or absolute import (tried pkg_resources._vendor.com,com).
Nuitka:WARNING:/usr/local/lib/python2.7/dist-packages/pkg_resources/_vendor/pyparsing.py:89: Cannot find 'ordereddict' in package 'pkg_resources._vendor' as relative or absolute import (tried pkg_resources._vendor.ordereddict,ordereddict).

Then, when I try to run examle.dist/example.exe I get following error ./example.exe

Traceback (most recent call last):
  File "/home/user/code/python/webrtc_vad/testing/py-webrtcvad/example.dist/example.py", line 6, in <module>
  File "/home/user/code/python/webrtc_vad/testing/py-webrtcvad/example.dist/webrtcvad.py", line 3, in <module webrtcvad>
ImportError: No module named _webrtcvad

How do I fix these WARNINGS and ERROR?

[kayhayen]

There we go, egg files are zip files:

Length Date Time Name

---

  536  2018-05-22 09:49   _webrtcvad.pyc
 1709  2018-05-22 09:49   webrtcvad.pyc
  976  2018-05-11 15:42   webrtcvad.py
  286  2018-05-22 09:49   _webrtcvad.py
163080  2018-05-22 09:49   _webrtcvad.so
   21  2018-05-22 09:49   EGG-INFO/top_level.txt
 2055  2018-05-22 09:49   EGG-INFO/SOURCES.txt
    1  2018-05-22 09:49   EGG-INFO/dependency_links.txt
   14  2018-05-22 09:49   EGG-INFO/native_libs.txt
    1  2018-05-22 09:49   EGG-INFO/zip-safe
 3739  2018-05-22 09:49   EGG-INFO/PKG-INFO
   74  2018-05-22 09:49   EGG-INFO/requires.txt

This is probably dealt with zipimporter, which might cache things in “.cache”, where you will then find it. In Nuitka there currently is nothing to support it, but it seems feasible to add.

These eggs are likely out of fashion, precisely because they require each user to privately extract on the fly. To check if an update is needed, some processing of the zip file will be needed each time, for every program you run, and that does an import not satisfied by earlier elements of the path. That will make your Python only slower.

So, no trick there inside the zip luckily. Please allow for some time to implement this. At least I now have an .egg file to test with. Hopefully it’s as easy as to ask zipimporter to do its thing. But could also be that it’s not possible to decouple searching from actual importing, then I get to implement the caching part myself. Ideally the cache hit is the normal case, and then it’s only about replacing the egg with a cache directory, and maybe complain if it’s not cached yet. Nuitka may not have to be the first to unpack an egg into a cache. But that is speculation, need to check what is there fore use.

Yours, Kay

[kayhayen]

So this is in the new pre-release now, closing now as release is immanent and not hotfixes will happen for 0.5.33 anymore.

For the pkg_resources issue I create #146 which seems possible, but will also not immediately be dealt with.