the test suite is too slow

The test suite is getting to be quite slow. Here are the 20 slowest tests on my mac:

pytest --durations=20

18.96s call     pvlib/tests/iotools/test_pvgis.py::test_get_pvgis_tmy_kwargs
16.32s call     pvlib/tests/iotools/test_psm3.py::test_get_psm3_singleyear
14.82s setup    pvlib/tests/test_forecast.py::test_process_data[HRRR]
8.12s call     pvlib/tests/iotools/test_psm3.py::test_get_psm3_tmy
6.82s setup    pvlib/tests/test_forecast.py::test_process_data[NDFD]
5.47s call     pvlib/tests/iotools/test_pvgis.py::test_get_pvgis_tmy_epw
4.78s call     pvlib/tests/iotools/test_pvgis.py::test_get_pvgis_tmy
3.99s setup    pvlib/tests/test_forecast.py::test_process_data[RAP]
3.45s call     pvlib/tests/iotools/test_srml.py::test_read_srml_month_from_solardat
3.20s call     pvlib/tests/test_forecast.py::test_bad_kwarg_get_data
3.16s call     pvlib/tests/test_forecast.py::test_bad_kwarg_get_processed_data
3.13s call     pvlib/tests/test_forecast.py::test_how_kwarg_get_processed_data
3.13s setup    pvlib/tests/test_forecast.py::test_process_data[NAM]
3.12s call     pvlib/tests/test_forecast.py::test_vert_level
3.08s call     pvlib/tests/test_forecast.py::test_datetime
2.98s call     pvlib/tests/test_irradiance.py::test_get_extra_radiation_nrel_numba
2.85s call     pvlib/tests/test_solarposition.py::test_spa_python_numba_physical
2.82s call     pvlib/tests/test_solarposition.py::test_get_solarposition_deltat[None-nrel_numba]
2.73s setup    pvlib/tests/test_spa.py::NumbaSpaTest::test_aberration_correction
2.69s call     pvlib/tests/iotools/test_pvgis.py::test_get_pvgis_tmy_basic

I will work on the forecast tests. Any ideas for the iotools tests?

[kanderso-nrel]

One idea is to mock the network calls by default [1] but leave the option to perform them when specifically requested [2].

[1] https://requests-mock.readthedocs.io/en/latest/pytest.html [2] https://docs.pytest.org/en/2.9.1/example/simple.html#control-skipping-of-tests-according-to-command-line-option

(edit – the [2] link above is super out of date, not sure why it showed up for me on google. a little irrelevant since pytest-remotedata does the same thing anyway)

[mikofski]

I’m -1 on splitting iotools into a separate package at this point, but open to it in the future. I think it’s too much overhead for now.

I was a little scared of that too, but just mentioned it as a remote possibility, nuclear option, thx

I’m in favor of running only the live network tests on one CI instance. I think that’s a common pattern for other pydata projects. I’m also in favor of mocking calls where appropriate.

I think a wiki or doc page in contribution guidelines (maybe I missed this) to clarify expected testing patterns and recommended PR test procedures could be helpful to align everyone