Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
When using standalone mode, limit the recursion into imported modules to only those modules that are actually utilized.
See original GitHub issueWhen trying to build a standalone executable, the number of included modules can quickly explode, especially if the application uses large third-party packages. Ideally, Nuitka should only recurse into the modules that are actually used by the application. Currently there is no mechanism to do so, apart from manually specifying --nofollow-import-to flags for all the unwanted modules.
As a simple example for motivation, suppose an application uses the popular machine learning package sklearn (AKA scikit-learn):
from sklearn import datasets
datasets.load_iris()
Even though the above code uses only load_iris(), Nuitka will simply look at the imports for the datasets module, which include the following:
from .base import load_breast_cancer
from .base import load_boston
from .base import load_diabetes
from .base import load_digits
from .base import load_files
from .base import load_iris
from .base import load_linnerud
from .base import load_sample_images
from .base import load_sample_image
from .base import load_wine
from .base import get_data_home
from .base import clear_data_home
from .covtype import fetch_covtype
from .kddcup99 import fetch_kddcup99
from .lfw import fetch_lfw_pairs
from .lfw import fetch_lfw_people
from .twenty_newsgroups import fetch_20newsgroups
from .twenty_newsgroups import fetch_20newsgroups_vectorized
from .mldata import fetch_mldata, mldata_filename
from .openml import fetch_openml
from .samples_generator import make_classification
from .samples_generator import make_multilabel_classification
from .samples_generator import make_hastie_10_2
from .samples_generator import make_regression
from .samples_generator import make_blobs
from .samples_generator import make_moons
from .samples_generator import make_circles
from .samples_generator import make_friedman1
from .samples_generator import make_friedman2
from .samples_generator import make_friedman3
from .samples_generator import make_low_rank_matrix
from .samples_generator import make_sparse_coded_signal
from .samples_generator import make_sparse_uncorrelated
from .samples_generator import make_spd_matrix
from .samples_generator import make_swiss_roll
from .samples_generator import make_s_curve
from .samples_generator import make_sparse_spd_matrix
from .samples_generator import make_gaussian_quantiles
from .samples_generator import make_biclusters
from .samples_generator import make_checkerboard
from .svmlight_format import load_svmlight_file
from .svmlight_format import load_svmlight_files
from .svmlight_format import dump_svmlight_file
from .olivetti_faces import fetch_olivetti_faces
from .species_distributions import fetch_species_distributions
from .california_housing import fetch_california_housing
from .rcv1 import fetch_rcv1
Nuitka will start to recurse into all of these modules, even though base.load_iris is the only one that is actually needed. Furthermore, it will recurse into hundreds of additional other modules after reading the import statements of the above scikit-learn modules.
Referencing a short conversation with @kayhayen on Gitter:
The idea here would be to use the import tracing hidden in hints.py in nuitka git checkout, run your program with it, collect all imports, and then to make a change to Nuitka, where --no-follow is added to the command line with --follow-to=all_the,names,you,saw,being,actually,imported That way, sort of making it possible to avoid packages that never even do get imported at least.
I will start to look into this technique. If others are interested to contribute, please let me know.
Issue Analytics
- State:
- Created 5 years ago
- Reactions:4
- Comments:25 (20 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Just finished a first version the stuff mentioned in my previous post. It consists of the following components:
hints.pyget-hints.pywhich reads the to-be-examined script and and runs it under control ofhints.py. It collects the output and creates a JSON filescript.jsonfor the Pythonscript.py. The JSON file is a sorted array of all imports made by the script during the test run. Duplicates have been removed.nuitka-hints.py, which invokes the Nuitka compiler, passing any appropriate or site / specific options to it (among them--standalone), and also--user-plugin=hinted-mods.py=script.json. Filescript.jsonis the file from the previous test run.hinted-mods.pyis a user plugin, which in its__init__reads the JSON file and adds some options in response to the JSON contents, like enabling plugins. Its main logic is contained in itsonModuleEncountermethod, where it checks for each module handed in by the Nuitka compiler, whether it is part of the JSON file.@robguinness, @kayhayen, @sannanansari: I am going to put a ZIP file of these components on the utilities repo. Maybe you are interested in trying something out yourself. Here is a sample output for a script that uses PIL / Pillow to open an image file:
@sannanansari , great that you can contribute! I’m not sure how quickly I can get to this, as it seems a bit more complicated than I originally thought. But I am still definitely interested in working on it. I will let you know if I make any progress.