Unit tests use up to 20GB of memory on circleci

Problem While debugging intermittent test failures on PR #1410, @christopherbunn and I measured memory usage of the unit tests on circleci and found a complete end-to-end run can use up to 20GB at peak.

That’s way more than I would have expected… the question is, why?

Observations We ssh’ed into a circleci box running on main and ran the following using memory-profiler:

mprof run  --include-children pytest evalml/ -n 8 --doctest-modules --cov=evalml --junitxml=test-reports/junit.xml --doctest-continue-on-failure -v

Which created the following plot, visible with mprof plot:

I ran this twice and got a similar plot, so the results appear to be consistent across runs.

This is dangerously close to the max memory allowed on the circleci worker size we’re using. That’s why we started looking into this – on #1410, we saw that the memory usage went 5GB higher for some reason.

[freddyaboulton]

We noticed that we can shave 1.5gb from just the automl tests (almost half!) by manually setting n_jobs=1 for all estimators used by automl (plots below). We verified that the value of n_jobs is a factor only in the few automl tests that don’t mock fit and score. Based on this, we have come up with the current plan:

1. for each component which accepts n_jobs as a parameter (i.e. the sklearn-based estimators), make sure we have one unit test which sets n_jobs=-1, to verify that works properly for that component.
2. for all other unit tests which don’t mock the underlying fit , set n_jobs=1 for all components to avoid memory and threading issues
3. make sure that in looking glass we’re running with n_jobs=-1, which i believe we currently are since the default value of n_jobs for relevant estimators is -1

Hopefully once this is done, we’ll see some nice improvements upon the overall memory footprint of the unit tests!

[gsheni]

@thehomebrewnerd @freddyaboulton We could have an inline import for the sklearn import (so it only runs when you call the mutual info function). I have seen us explicitly do this in a few in other libraries. We generally do it to avoid circular imports. It would feel weird to do it just to save memory…