memory leak in MultiIndex

Code Sample, a copy-pastable example if possible

# Your code here
import pandas as pd
import numpy as np
import gc
import psutil

def totmem(p):
    # total memory used by process in MB
    info = p.memory_info()
    return 1e-6*(info.vms + info.rss)

dat_size = 6000000
# make dataset with no data
# uncomment for regular index
# dat = pd.DataFrame(index=np.arange(dat_size))
# uncomment for MultiIndex index
dat = pd.DataFrame(index=pd.MultiIndex.from_arrays((np.arange(dat_size), np.arange(dat_size))))
# make bool vector for subsetting
sub = np.ones(dat_size, dtype=bool)
# init psutil stuff
p = psutil.Process()
gc.collect()
ram = totmem(p)

for i in range(10):
    dat.iloc[sub, :]     # leak happens here
    gc.collect()
    print(int(totmem(p) - ram))
    ram = totmem(p)

Problem description

this program leaks at a rate of 191 MB / cycle; it eventually runs out of memory if the loop goes on indefinitely. In the program we use a bool vector to subset a dataset. If the dataset index is MultiIndex we observe a memory leak (as reported by the print statement). If the index is a regular mono-index no such leak is observed.

Expected Output

• the program should not run out memory.
• every output line should be zero.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line] INSTALLED VERSIONS

python: 3.6.6.final.0 python-bits: 64 OS: Linux pandas: 0.23.4 numpy: 1.15.1

[mroeschke]

Numpy’s minimum version will be 1.17.3 in the next release (1.3) which shouldnt have this bug. Going to close but happy to reopen if this resurfaces

[Meta95]

Can anyone confirm this issue is fixed for numpy 0.15.3 and above?

Update: I did an install of 0.15.3. OP’s script still reports the same memory leak. Will try some other versions of numpy and see if it got fixed anywhere.

Update 2: Turns out my environment was a mess. A proper upgrade to 0.15.3 does indeed solve the issue! My pandas version is 0.23.4 and python version 3.6.6 like the others up above. Please feel free to verify yourselves and close this ticket.