TST, CI: Azure Windows Run Time Diffs

This is primarily intended as a concise report of some Azure CI behavior to @chrispat and @vtbassmatt.

In short, it seems that when I run effectively identical branches in the main SciPy repo vs. in my fork of the repo (with effectively identical target master branches), the timings for running our test suite are vastly different. Could it be that there’s a priority algorithm directing the jobs in the heavily-used repo to different hardware than the repo that is almost never used?

Here’s some summary data comparing the two scenarios:

• PR on my fork where 32-bit Windows job finishes in 41 minutes, 46 seconds: https://github.com/tylerjereddy/scipy/pull/29
• effectively identical PR to main repo where 32-bit Windows job finishes in 52 minutes, 19 seconds: https://github.com/scipy/scipy/pull/11694

Since the Azure logs get purged every ~10 days or so, here is the raw text output for:

• my fork PR Windows 32-bit log: tyler_fork_log.txt

• main repo PR Windows 32-bit log: main_repo_log.txt

The big difference seems to be in the amount of time required to run the full test suite; the slowest 10 test durations suggest that there may be some heterogeneity in assigned hardware:

Fork Windows 32-bit Win slowest 10 tests:

2020-03-21T15:30:10.7264434Z ========================== slowest 10 test durations ==========================
2020-03-21T15:30:10.7286762Z 138.80s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_p_zero
2020-03-21T15:30:10.7291924Z 88.02s call     optimize/tests/test__differential_evolution.py::TestDifferentialEvolutionSolver::test_L8
2020-03-21T15:30:10.7292840Z 72.08s call     sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py::test_diagonal_data_types
2020-03-21T15:30:10.7294146Z 63.72s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_new_method
2020-03-21T15:30:10.7321531Z 49.00s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_without_mode_shift
2020-03-21T15:30:10.7322497Z 38.24s call     stats/tests/test_continuous_basic.py::test_cont_basic[kstwo-arg59]
2020-03-21T15:30:10.7323293Z 36.45s call     special/tests/test_mpmath.py::TestSystematic::test_legenp_complex_3
2020-03-21T15:30:10.7324067Z 35.74s call     special/tests/test_mpmath.py::TestSystematic::test_legenp_complex_2
2020-03-21T15:30:10.7324855Z 34.31s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_with_mode_shift
2020-03-21T15:30:10.7325610Z 33.23s call     stats/tests/test_stats.py::TestMGCStat::test_twosamp

Main Repo 32-bit Win slowest 10 tests:

2020-03-21T15:35:21.7974905Z ========================== slowest 10 test durations ==========================
2020-03-21T15:35:21.7995869Z 176.72s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_p_zero
2020-03-21T15:35:21.7997603Z 125.06s call     sparse/linalg/tests/test_expm_multiply.py::TestExpmActionInterval::test_expm_multiply_interval_matrix
2020-03-21T15:35:21.7999866Z 112.96s call     sparse/linalg/eigen/lobpcg/tests/test_lobpcg.py::test_diagonal_data_types
2020-03-21T15:35:21.8001497Z 104.41s call     sparse/linalg/tests/test_matfuncs.py::TestExpM::test_pascal
2020-03-21T15:35:21.8002826Z 103.77s call     optimize/tests/test__differential_evolution.py::TestDifferentialEvolutionSolver::test_L8
2020-03-21T15:35:21.8004589Z 78.71s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_new_method
2020-03-21T15:35:21.8006064Z 69.11s call     sparse/linalg/tests/test_expm_multiply.py::TestExpmActionSimple::test_expm_multiply
2020-03-21T15:35:21.8008332Z 55.78s call     stats/tests/test_distributions.py::TestGenInvGauss::test_rvs_without_mode_shift
2020-03-21T15:35:21.8009880Z 47.74s call     stats/tests/test_continuous_basic.py::test_cont_basic[kstwo-arg59]
2020-03-21T15:35:21.8011662Z 45.46s call     stats/tests/test_stats.py::TestMGCStat::test_twosamp

[sethtroisi]

I got tests running in colab and I plotted the tests which vary most between the two https://colab.research.google.com/drive/1GLYz5AK6G1PjAJjsRvIsXL5-rDlBfgK_#scrollTo=jW_sVHK0wDE5

UPDATED with test names

If someone provides me with a pointer to an Azure run with --durations=0 (or 100 or 1000) I’ll computer the same for Azure

[andyfaff]

Back to the main topic, I can understand why we still run CI tests for Linux32 (embedded boards, etc). What’s the rationale for keeping the windows32 bit tests?