improve rdtools import time

RdTools is a little slow to import:

In [1]: %time import rdtools
Wall time: 1.04 s

For comparison:

In [1]: %time import pvlib
Wall time: 184 ms

Here’s a breakdown of where import rdtools is spending time:

Generated with:

(base) C:\Windows\Temp>python -X importtime -c "import rdtools" 2> rdtools.log
(base) C:\Windows\Temp>tuna rdtools.log

We could speed up the import time by changing how we import packages. I notice that statsmodels.api, pkg_resources, and h5py together make up a large chunk of the total but aren’t actually used in the “primary” RdTools functions. What do people think about moving those imports into the functions that need them instead of importing at module scope?

Pros:

• It decreases import time to 600 ms (~40% speedup) on my machine
• It makes those dependencies optional for people that don’t plan on using those functions – e.g. our fleets pipeline doesn’t actually need the statsmodels package because it doesn’t use the OLS and classical decomp functions.

Cons:

• It’s nice to have a list of imports at the top of the module and hiding them in the functions is nonstandard and reduces code clarity
• It violates pep8 (https://www.python.org/dev/peps/pep-0008/#imports)
• It makes the first function invocation slower, and subsequent invocations very marginally slower
• 1 second to import isn’t that big a deal

[cwhanse]

Good point, though that’s an actual change instead of just minor restructuring. Maybe np.polyfit?

My $0.02: unless importing statsmodels becomes a performance headache, I’d stick with it for the long term. There are regression functions in numpy (numpy.linalg.lstsq) but my sense is that the long-term intent is to provide regression and similar functions in statsmodels and scipy, with numpy providing the building blocks. And statsmodels provides options such as robust regression that may become desireable for RdTools applications.

[mdeceglie]

This is almost a year old and the marginal gains probably don’t justify violating style guides to put imports in functions. I’ll close this for now.