Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

run profiling to determine loading times

See original GitHub issue

We need to optimize the time it takes ploomber to load when running ploomber or ploomber --help

We need to run profiling and see the different load times of the python packages and how we can optimize it to get a faster user response.

Issue Analytics

State:
Created 2 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

1reaction

edublancascommented, Feb 6, 2022

this output looks like C-level modules, we’d like more general information. this looks useful: https://pypi.org/project/import-profiler/

0reactions

Sri0027commented, Feb 6, 2022

Using cProfile, I got the following information on top 10 calls taking most time. ncalls tottime percall cumtime percall filename:lineno(function) 223 0.008 0.000 14.722 0.066 init.py:1(<module>) 2376/1 0.029 0.000 3.715 3.715 {built-in method builtins.exec} 2339/1 0.022 0.000 3.715 3.715 <frozen importlib._bootstrap>:986(_find_and_load) 2331/1 0.013 0.000 3.715 3.715 <frozen importlib._bootstrap>:956(_find_and_load_unlocked) 2218/1 0.013 0.000 3.714 3.714 <frozen importlib._bootstrap>:650(_load_unlocked) 2085/1 0.009 0.000 3.714 3.714 <frozen importlib._bootstrap_external>:777(exec_module) 2755/1 0.003 0.000 3.714 3.714 <frozen importlib._bootstrap>:211(_call_with_frames_removed) 3 0.000 0.000 3.672 1.224 dag.py:1(<module>) 790/48 0.003 0.000 3.482 0.073 {built-in method builtins.import} 2423/890 0.005 0.000 1.958 0.002 <frozen importlib._bootstrap>:1017(_handle_fromlist)

On using an import filter, it gives the following.

ncalls tottime percall cumtime percall filename:lineno(function) 2339/1 0.022 0.000 3.715 3.715 <frozen importlib._bootstrap>:986(_find_and_load) 2331/1 0.013 0.000 3.715 3.715 <frozen importlib._bootstrap>:956(_find_and_load_unlocked) 2218/1 0.013 0.000 3.714 3.714 <frozen importlib._bootstrap>:650(_load_unlocked) 2085/1 0.009 0.000 3.714 3.714 <frozen importlib._bootstrap_external>:777(exec_module) 2755/1 0.003 0.000 3.714 3.714 <frozen importlib._bootstrap>:211(_call_with_frames_removed) 790/48 0.003 0.000 3.482 0.073 {built-in method builtins.import} 2423/890 0.005 0.000 1.958 0.002 <frozen importlib._bootstrap>:1017(_handle_fromlist)

Does this make sense?