ModuleNotFoundError for Pandas on standalone build

I’ve tried compiling a standalone application with a pandas==0.23.4 dependency, which gives a ModuleNotFoundError: No module named 'pandas._libs.tslibs.np_datetime' exception on running.

This section of the pandas code does seem to also be cpython extensions / compiled *.so’s. Also for reference, this is the line it raises on.

I’m unsure if nuitka should be auto descovering these kind of dependancies or not, I’ve seen similar things in pyinstaller. I did try adding variations of the argument --include-module=pandas._libs.tslibs.np_datetime to the build command, was unsuccessful. Though looking in the

This might just be user error, though building this as a compiled module did not seem to have this problem, as I mentioned in #164.

Here’s the relevant code snippets:

0.6.0.3
Python: 3.7.0 (default, Sep  5 2018, 03:25:31) 
Executable: /usr/local/bin/python
OS: Linux
Arch: x86_64€

$ pip install nuitka==0.6.0.3 pandas==0.23.4`

# main.py
import pandas as pd
print(pd.Series([1, 2, 3]))

$ python -m nuitka main.py --standalone
$ main.dist/main
Traceback (most recent call last):
  File ".../main.dist/pandas/__init__.py", line 26, in <module pandas>
  File ".../main.dist/pandas/_libs/__init__.py", line 4, in <module pandas._libs>
  File "pandas/_libs/tslibs/conversion.pxd", line 11, in init pandas._libs.tslib
  File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
ModuleNotFoundError: No module named 'pandas._libs.tslibs.np_datetime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../main.dist/__main__.py", line 1, in <module>
  File ".../main.dist/playground/__init__.py", line 1, in <module playground>
  File ".../main.dist/playground/pandas_.py", line 1, in <module playground.pandas_>
  File ".../main.dist/pandas/__init__.py", line 32, in <module pandas>
ImportError: C extension: No module named 'pandas._libs.tslibs.np_datetime' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

[oz123]

This is still an issue with 0.6.5. Maybe it’s me being noob … I did the following:

$ nuitka3 --version
0.6.5
Python: 3.6.5 (default, Jan 27 2019, 14:36:53) 
Executable: /usr/bin/python3.6
OS: Linux
Arch: x86_64
$ cat test.py 
import pandas as pd
print(pd.Series([1, 2, 3]))

Compile

nuitka3 --follow-imports --recurse-all  --show-progress test.py 
Nuitka:INFO:PASS 1:
Nuitka:INFO:Initial optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:Next global optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:PASS 2 ... :
Nuitka:INFO:Next global optimization pass.
Nuitka:INFO:Optimizing module '__main__', 0 more modules to go after that.
Nuitka:INFO:Doing module local optimizations for '__main__'.
Nuitka:INFO:Total memory usage before generating C code: 29.86 MB (31305728 bytes):
Nuitka:INFO:Total memory usage before running scons: 30.11 MB (31576064 bytes):
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libm.a when searching for -lm
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc

Run the binary

$ ./test.bin 
Traceback (most recent call last):
  File "/home/oznn/Software/cloudpoint-vsphere-exporter-schema/test.py", line 1, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

It would be really great if this repository contained a few examples on how to compile non tivial python code with some popular libraries like opencv, PIL and numpy.

(The test suite seems awesome but it requires sometime to figure out …)

Update

I found out that if install nuitka in my virtualenv and run it again and run it in the following way:

python -m nuitka --follow-imports --recurse-all  --show-progress test.py

Now the situation is a bit better

$ ./test.bin
Traceback (most recent call last):
  File "/home/oznn/.cache/pypoetry/virtualenvs/poetry-demo-py3.6/lib64/python3.6/site-packages/pandas/__init__.py", line 30, in <module pandas>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
ImportError: cannot import name 'hashtable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/oznn/Software/cloudpoint-vsphere-exporter-schema/test.py", line 1, in <module>
    import pandas as pd
  File "/home/oznn/.cache/pypoetry/virtualenvs/poetry-demo-py3.6/lib64/python3.6/site-packages/pandas/__init__.py", line 34, in <module pandas>
    raise ImportError(
ImportError: C extension: 'hashtable' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

[JorjMcKie]

Hi @oz123 - I know you to be a PyMuPDF stargazer since a long time 😉

It would be really great if this repository contained a few examples on how to compile non tivial python code with some popular libraries like opencv, PIL and numpy.

I recommend you use “hinted compilation” for standalone builds. As pointed out there, any required standard plugins are automatically included, and you in addition should end up with a smaller dist folder after a shorter compilation time. Let me repeat a few highlights here:

• Only suitable for standalone builds
• Consists of two separate steps:
  1. Run your script “under observation” of a logger script (get-hints.py), which records all imports actually executed.
  2. Run a modified compile script python nuitka-hints.py yourscript.py. As you see, this command line is very short. This is because the log file created in previous step is used to generated any additionally required parameters.