Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Performance measurement helpers and examples

See original GitHub issue

Feature request

For a library that is used to speed up the code, it seems quite essential to have an easy way to measure the result of modifications. Both for improvements, for regressions, and just for testing how things work. However, even the Performance Tips doesn’t contain the code to repeat the performance measurements.

As a DevOps who deals with updating dependencies almost on a daily basis, because of security issues or build failures, I would appreciate if documentation contained examples with code snippets and provided helpers to profile and compare Numba performance between releases.

I found that people are using timeit() with custom formatters for the results.

From timeit execution it it not really clear how much time was spent on compiling the code, if anything was compiled at all, if the compiled code was faster of slower.

It looks like people mostly do the measurements manually, adding and removing decorators, restarting notebooks for each measurement. I would expect something more automated, maybe a modification to @jit decorator that will add some profiling mode to it, where it will be possible to rerun wrapped functions with timeit() and collect the data in a format suitable for comparison, with distinction between compile time and run time, maybe even analysing the data passed to functions to detect data specific problems. This behaviour in this profiling mode could then be controlled by environment variables.

Ideally as a DevOps I just want a single command to get and measure performance, which will be added into CI/CD cycle. Or at least a clear instruction how to get the fixture, where to place the files, how to run the measurement and compare the results. An instruction for a person with a zero knowledge of Numba would be ideal, meaning that it could be followed by anyone who would like to save me some time for other, no less important tasks.