Very high memory usage on Windows 10

Reproducing code example:

from memory_profiler import profile

def process(num):
    result = 0;
    for i in range(num):
        result += i**2

@profile
def main():
    from scipy import integrate
    process(5000000)
    print("Done")

if __name__ == '__main__':
    main()

output

Done
Filename: multiprocess.py

Line #    Mem usage    Increment   Line Contents
================================================
     8     33.0 MiB     33.0 MiB   @profile
     9                             def main():
    10     57.6 MiB     24.5 MiB       from scipy import integrate
    11     57.6 MiB      0.0 MiB       process(5000000)
    12     57.6 MiB      0.0 MiB       print("Done")

In Windows Task Manager > Details tab > right click a column header > Select columns > Commit size Run the above script Watch the commit size of the new python process in Task Manager Note 1.5GB commit size (while working set is not that large, and @profile reports only 57MB.

When from scipy import integrate is commented out, the commit size will be only 33MB.

This high memory commitment is a problem especially when running many processes in parallel. Ex: https://github.com/mwaskom/seaborn/issues/2181

Scipy/Numpy/Python version information:

import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)

1.2.0 1.18.4 sys.version_info(major=3, minor=6, micro=5, releaselevel='final', serial=0)

[ragibson]

I’m not sure why the first interpreter over-commits so heavily, it does not appear significantly different than the second. I suppose you could try compiling your own version of CPython.

It looks like that on Windows, multiprocessing initializes a completely new interpreter for each child process. This means that your imports are actually repeated in each child rather than just sharing the imported code in the memory of the parent.

From my perspective, you have a few options

1. Increase the size of your page file. I’m not sure why you have it disabled, but it is generally a misconception to believe that this would increase performance.
2. Decrease the number of child processes you are using. It seems your system is hyperthreaded, so you might not lose much performance by cutting the number of children in half (depending on the workload of interest).
3. Find an interpreter that doesn’t commit so much memory. I honestly have no idea why there is so much variation between the two interpreters on my Windows machine – I do most of my development on Linux and don’t have the tools to figure this out at the moment.
4. Add additional memory to your machine.
5. Switch to an OS with more efficient multiprocessing/forking semantics.

[rgommers]

Part of the issue here will be with OpenBLAS allocating ~30 MB per thread. So on a 32-core machine that’ll take about 1 GB. This was improved as much as possible in recent OpenBLAS versions. You can also try to limit the number of threads it uses (https://github.com/xianyi/OpenBLAS#setting-the-number-of-threads-using-environment-variables).

Please reply with what works or doesn’t work and the OpenBLAS version you are using, if you are still seeing this problem.

If you use conda you can also switch to MKL, which doesn’t allocate memory like this.