Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Datashader import extremely slow
See original GitHub issueIt looks like datashader is now compiling a bunch of stuff on import which makes the import extremely slow. Here’s what I get when I run:
%%prun -s cumulative
__import__('datashader')
. 17488551 function calls (16493744 primitive calls) in 11.447 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
101 0.002 0.000 23.415 0.232 __init__.py:1(<module>)
1838/1 0.011 0.000 11.454 11.454 {built-in method builtins.exec}
817/1 0.001 0.000 11.454 11.454 {built-in method builtins.__import__}
1643/1 0.008 0.000 11.454 11.454 <frozen importlib._bootstrap>:978(_find_and_load)
1633/1 0.005 0.000 11.454 11.454 <frozen importlib._bootstrap>:948(_find_and_load_unlocked)
1501/1 0.006 0.000 11.452 11.452 <frozen importlib._bootstrap>:663(_load_unlocked)
1370/1 0.004 0.000 11.452 11.452 <frozen importlib._bootstrap_external>:722(exec_module)
2031/1 0.001 0.000 11.452 11.452 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
17 0.001 0.000 10.644 0.626 core.py:1(<module>)
43/39 0.000 0.000 9.436 0.242 compiler.py:502(compile_extra)
825/127 0.002 0.000 9.354 0.074 compiler_lock.py:29(_acquire_compile_lock)
43/39 0.000 0.000 9.302 0.239 compiler.py:313(compile_extra)
43/39 0.000 0.000 9.285 0.238 compiler.py:380(_compile_bytecode)
43/39 0.000 0.000 9.285 0.238 compiler.py:343(_compile_core)
43/39 0.002 0.000 9.278 0.238 compiler_machinery.py:305(run)
774/702 0.006 0.000 9.270 0.013 compiler_machinery.py:263(_runPass)
2322/2106 0.003 0.000 9.261 0.004 compiler_machinery.py:267(check)
47/38 0.001 0.000 8.623 0.227 dispatcher.py:744(compile)
38/35 0.000 0.000 8.616 0.246 dispatcher.py:76(compile)
38/35 0.000 0.000 8.616 0.246 dispatcher.py:83(_compile_cached)
38/35 0.003 0.000 8.616 0.246 dispatcher.py:98(_compile_core)
225/212 0.001 0.000 8.351 0.039 decorators.py:184(wrapper)
1 0.000 0.000 8.295 8.295 viewshed.py:1(<module>)
43 0.001 0.000 6.516 0.152 typed_passes.py:369(run_pass)
43 0.002 0.000 6.506 0.151 typed_passes.py:304(run_pass)
43 0.000 0.000 4.611 0.107 lowering.py:174(lower)
43 0.000 0.000 3.179 0.074 lowering.py:212(lower_normal_function)
43 0.001 0.000 3.153 0.073 lowering.py:226(lower_function_body)
582 0.019 0.000 3.148 0.005 lowering.py:250(lower_block)
7126 0.028 0.000 3.079 0.000 lowering.py:297(lower_inst)
79222/69385 0.173 0.000 2.522 0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)
21311/21309 2.046 0.000 2.049 0.000 ffi.py:112(__call__)
2995 0.006 0.000 1.907 0.001 lowering.py:455(lower_assign)
1443 0.008 0.000 1.838 0.001 lowering.py:901(lower_expr)
49 0.001 0.000 1.647 0.034 codegen.py:196(finalize)
98 0.000 0.000 1.637 0.017 codegen.py:459(get_pointer_to_function)
344/245 0.000 0.000 1.636 0.007 codegen.py:102(_ensure_finalized)
87 0.001 0.000 1.595 0.018 codegen.py:178(add_ir_module)
1467/1456 0.003 0.000 1.546 0.001 base.py:1131(__call__)
1467/1456 0.002 0.000 1.538 0.001 base.py:1155(wrapper)
45 0.001 0.000 1.514 0.034 cpu.py:165(get_executable)
43/39 0.001 0.000 1.472 0.038 typed_passes.py:82(run_pass)
43/39 0.001 0.000 1.470 0.038 typed_passes.py:46(type_inference_stage)
2 0.000 0.000 1.441 0.720 points.py:1(<module>)
43/39 0.000 0.000 1.391 0.036 typeinfer.py:932(propagate)
87/79 0.011 0.000 1.388 0.018 typeinfer.py:135(propagate)
5 0.000 0.000 1.329 0.266 dufunc.py:168(_compile_for_argtys)
591 0.005 0.000 1.273 0.002 lowering.py:591(lower_getitem)
5 0.000 0.000 1.157 0.231 ufuncbuilder.py:127(_compile_element_wise_function)
5 0.000 0.000 1.157 0.231 ufuncbuilder.py:63(compile)
5 0.000 0.000 1.157 0.231 ufuncbuilder.py:81(_compile_core)
2509/2445 0.014 0.000 1.128 0.000 typeinfer.py:467(resolve)
591 0.002 0.000 1.094 0.002 arrayobj.py:410(getitem_arraynd_intp)
3485/3411 0.003 0.000 1.094 0.000 context.py:192(resolve_function_type)
258 0.005 0.000 1.074 0.004 postproc.py:70(run)
10 0.000 0.000 1.069 0.107 __init__.py:3(<module>)
2511/2447 0.002 0.000 1.055 0.000 typeinfer.py:1318(resolve_call)
3485/3411 0.008 0.000 1.052 0.000 context.py:236(_resolve_user_function_type)
533 0.002 0.000 1.028 0.002 module.py:235(__repr__)
533 0.001 0.000 1.021 0.002 module.py:209(_get_body_lines)
533 0.002 0.000 1.019 0.002 module.py:214(<listcomp>)
470076/200726 0.261 0.000 1.010 0.000 {method 'format' of 'str' objects}
825 0.000 0.000 1.003 0.001 values.py:668(__str__)
825 0.000 0.000 0.999 0.001 values.py:661(descr)
93 0.001 0.000 0.967 0.010 values.py:654(descr_body)
2732 0.004 0.000 0.966 0.000 values.py:795(descr)
2732 0.016 0.000 0.957 0.000 values.py:797(<listcomp>)
306576/124122 0.133 0.000 0.947 0.000 _utils.py:46(__str__)
3801 0.075 0.000 0.927 0.000 functions.py:122(get_call_type)
258 0.001 0.000 0.925 0.004 postproc.py:153(_insert_var_dels)
2 0.000 0.000 0.863 0.431 hilbert_curve.py:1(<module>)
1 0.000 0.000 0.861 0.861 decorators.py:117(wrap)
1 0.000 0.000 0.851 0.851 dufunc.py:133(add)
86661 0.134 0.000 0.824 0.000 values.py:223(_to_string)
591 0.001 0.000 0.820 0.001 arrayobj.py:390(_getitem_array_generic)
49 0.000 0.000 0.798 0.016 codegen.py:120(_optimize_final_module)
51 0.000 0.000 0.787 0.015 codegen.py:242(_finalize_final_module)
51 0.000 0.000 0.758 0.015 codegen.py:480(_finalize_specific)
191 0.000 0.000 0.748 0.004 codegen.py:542(wrapper)
51 0.000 0.000 0.746 0.015 executionengine.py:90(finalize_object)
2191 0.006 0.000 0.743 0.000 typeinfer.py:571(__call__)
41081/41048 0.041 0.000 0.700 0.000 templates.py:232(apply)
49 0.000 0.000 0.697 0.014 passmanagers.py:94(run)
1079/303 0.001 0.000 0.617 0.002 utils.py:387(__get__)
85671 0.273 0.000 0.612 0.000 instructions.py:14(__init__)
258 0.001 0.000 0.586 0.002 postproc.py:53(deadmaps)
767 0.008 0.000 0.534 0.001 arrayobj.py:315(basic_indexing)
86 0.012 0.000 0.524 0.006 registry.py:56(apply)
16 0.001 0.000 0.507 0.032 utils.py:1(<module>)
1 0.000 0.000 0.497 0.497 pipeline.py:1(<module>)
1 0.000 0.000 0.486 0.486 composite.py:5(<module>)
4 0.000 0.000 0.479 0.120 composite.py:48(operator)
432482/94149 0.324 0.000 0.474 0.000 ir.py:306(_rec_list_vars)
3299586 0.357 0.000 0.466 0.000 {built-in method builtins.isinstance}
85497 0.022 0.000 0.457 0.000 ir.py:339(list_vars)
318/298 0.002 0.000 0.452 0.002 typeinfer.py:457(__call__)
2995 0.008 0.000 0.445 0.000 lowering.py:1135(storevar)
87 0.000 0.000 0.444 0.005 codegen.py:190(add_llvm_module)
197/191 0.001 0.000 0.422 0.002 functions.py:268(get_call_type)
197/191 0.001 0.000 0.417 0.002 dispatcher.py:286(get_call_template)
258 0.003 0.000 0.414 0.002 postproc.py:47(livemap)
9001 0.020 0.000 0.401 0.000 context.py:199(_call_incref_decref)
4650/3057 0.017 0.000 0.398 0.000 base.py:520(get_function)
176 0.002 0.000 0.391 0.002 lowering.py:418(lower_setitem)
312 0.002 0.000 0.377 0.001 arrayobj.py:374(make_view)
1370 0.014 0.000 0.375 0.000 <frozen importlib._bootstrap_external>:793(get_code)
38 0.000 0.000 0.364 0.010 lowering.py:262(create_cpython_wrapper)
38 0.000 0.000 0.364 0.010 cpu.py:154(create_cpython_wrapper)
1252 0.010 0.000 0.361 0.000 cgutils.py:99(__init__)
94478 0.072 0.000 0.350 0.000 values.py:216(__init__)
1635 0.008 0.000 0.341 0.000 base.py:257(refresh)
87 0.002 0.000 0.340 0.004 codegen.py:106(_optimize_functions)
258 0.092 0.000 0.337 0.001 postproc.py:173(_patch_var_dels)
21867/9001 0.046 0.000 0.334 0.000 context.py:182(get_meminfos)
185218/174283 0.045 0.000 0.334 0.000 {built-in method builtins.next}
2732 0.006 0.000 0.334 0.000 npydecl.py:201(generic)
9941 0.012 0.000 0.330 0.000 cgutils.py:817(gep_inbounds)
2742 0.017 0.000 0.320 0.000 npydecl.py:103(generic)
9942 0.029 0.000 0.319 0.000 cgutils.py:824(gep)
825 0.001 0.000 0.315 0.000 passmanagers.py:123(run)
1165 0.004 0.000 0.313 0.000 arrayobj.py:69(_make_refs)
258 0.000 0.000 0.306 0.001 postproc.py:43(usedefs)
258 0.048 0.000 0.306 0.001 analysis.py:23(compute_use_defs)
43 0.000 0.000 0.295 0.007 untyped_passes.py:71(run_pass)
43 0.003 0.000 0.295 0.007 interpreter.py:85(interpret)
645 0.010 0.000 0.294 0.000 analysis.py:235(compute_cfg_from_blocks)
767 0.005 0.000 0.290 0.000 cgutils.py:659(get_item_pointer)
6655 0.003 0.000 0.289 0.000 context.py:222(decref)
176 0.001 0.000 0.289 0.002 arrayobj.py:452(setitem_array)
4507 0.018 0.000 0.287 0.000 cgutils.py:362(alloca_once)
447 0.008 0.000 0.281 0.001 lowering.py:523(lower_binop)
688 0.003 0.000 0.280 0.000 controlflow.py:110(process)
...
Issue Analytics
- State:
- Created 4 years ago
- Reactions:4
- Comments:10 (8 by maintainers)
Top Results From Across the Web
Geoviews + Datashader is slow when projecting points
First setup some data. import numpy as np import pandas as pd import dask.dataframe as dd ...
Read more >First calls to canvas.line are always incredibly slow - Datashader
When calling canvas.line , the first call always takes about 1 second no matter how many point I plot, which is very slow....
Read more >Performance — Datashader v0.14.3
Datashader is designed to make it simple to work with even very large datasets. To get good performance, it is essential that each...
Read more >Bokeh is too slow, please help - Community Support
I have tried to run the following code on JupyterLab with Python 3.8 and Bokeh 2.1.1, but it's extremely slow and totally un-usable....
Read more >Working with large data using datashader — HoloViews v1.15.3
import datashader as ds import numpy as np import holoviews as hv import pandas ... So even for very large datasets, you can...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
We’re expecting to make a release on Friday (tomorrow) that fixes import times, though unexpected calamities could delay it…
In case others need a workaround for the time being, datashader 0.8.0 and 0.7.0 import faster: cProfile timings for a panel application of mine (that also imports holoviews & datashader):
Of course, the slow-down occurs only once upon import, but it means debugging applications via
panel/bokeh servebecomes very slow. Even 7s is a long time for this, though, so I guess one should only ever debug datashader applications in jupyter notebooks (?).