Use RSS or VMS to track memory?

We recently added mechanisms to track a worker’s memory use in order to slow down or terminate that worker to avoid out-of-memory errors. When tracking memory we can look at either RSS or VMS memory use.

[1]: import psutil

In [2]: psutil.Process().memory_info()
Out[2]: pmem(rss=34463744, vms=289349632, shared=5316608, text=4096, lib=0, data=179593216, dirty=0)

Which should we be tracking? What do job schedulers track?

VMS is a bit unfortunate in that it includes 300MB of Python and 600MB including numpy, pandas, etc. which are sometimes just forked from the main process.

RSS is a bit unfortunate in that it seems not to include data allocated by NumPy.

In [3]: import numpy as np

In [4]: psutil.Process().memory_info()
Out[4]: pmem(rss=43642880, vms=358273024, shared=8069120, text=4096, lib=0, data=186109952, dirty=0)

In [5]: x = np.empty(int(1e9), dtype='u1')

In [6]: psutil.Process().memory_info()
Out[6]: pmem(rss=43642880, vms=1358274560, shared=8069120, text=4096, lib=0, data=1186111488, dirty=0)

Given this I think we have to use VMS. I’m not very familiar with what is commonly tracked by job schedulers that might kill a job though, so I thought I’d put this up here for general comment.

cc @davidedelvento

[mrocklin]

OK, so my case of not seeing RSS rise was more a function of my use of empty (or zeros it turns out) rather than a real use case.

In [1]: import psutil

In [2]: import numpy as np

In [3]: psutil.Process().memory_info()
Out[3]: pmem(rss=43540480, vms=366620672, shared=8032256, text=4096, lib=0, data=194457600, dirty=0)

In [4]: x = np.zeros(int(1e9), dtype='u1')

In [5]: psutil.Process().memory_info()
Out[5]: pmem(rss=43839488, vms=1366884352, shared=8069120, text=4096, lib=0, data=1194721280, dirty=0)

In [6]: y = np.ones(int(1e9), dtype='u1')

In [7]: psutil.Process().memory_info()
Out[7]: pmem(rss=1043877888, vms=2366885888, shared=8081408, text=4096, lib=0, data=2194722816, dirty=0)

In [8]: z = np.ones(int(1e9), dtype='u1')

In [9]: psutil.Process().memory_info()
Out[9]: pmem(rss=2043871232, vms=3366887424, shared=8081408, text=4096, lib=0, data=3194724352, dirty=0)

[seibert]

This may be the explanation:

  Initially, an anonymous mapping only allocates virtual memory.  The new
  mapping starts with a redundant copy on write mapping of the zero page.
  (The zero page is a single page of physical memory filled with zeroes,
  maintained by the operating system.)  Every virtual page of the anonymous
  mapping is attached to this existing prezeroed page, so attempts to read
  from anywhere in the mapping return zeroed memory even though no new
  physical memory has been allocated to it yet.

  Attempts to write to the page trigger the normal copy-on-write mechanism in
  the page fault handler, allocating fresh memory only when needed to allow
  the write to proceed.  (Note, prezeroing optimizations change the
  implementation details here, but the theory's the same.)  Thus "dirtying"
  anonymous pages allocates physical memory, the actual allocation call only
  allocates virtual memory.

From: https://landley.net/writing/memory-faq.txt