Memory leak while having spyder run too much stuff or for too long.

Description

What steps will reproduce the problem?

Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.

Versions

• Spyder version: 4.2.1
• Python version: 3.7.9
• Qt version: 5.12.10
• PyQt5 version: 5.12.3
• Operating System: Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree                    :  None (OK)
IPython >=7.6.0                 :  7.19.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  21.5.0 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.7.3 (OK)
pygments >=2.0                  :  2.7.3 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.1 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  51.0.0 (OK)
sphinx >=0.6.6                  :  3.3.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.1 (OK)
textdistance >=4.2.0            :  4.2.0 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog                        :  1.0.2 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.3 (OK)
numpy >=1.7                     :  1.19.3 (OK)
pandas >=1.1.1                  :  1.1.5 (OK)
scipy >=0.17.0                  :  1.5.4 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

[ccordoba12]

@stevetracvc, that doesn’t seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes.

Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder.

[stevetracvc]

@ccordoba12 the variable explorer didn’t seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as

while True:
    print("all work and no play makes Jack a dull boy")

does NOT replicate it, so it must be something more complicated.

Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:

{'fig': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x06\xaa\x00\x00\x02\x99\x08\x06\x00\x00\x00\xc0\xb2z\xeb\x00\x00\x009tEXtSoftware\x00Matplotlib version3.5.1, https://matplotlib.org/\xd8a\xf2\xbd\x00\x00\x00\tpHYs\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00\xf1\xfeIDATx\x9c\xec\xddwX\x14\xd7\xfb>\xfe\x9b&\x02\x0b,M\x01A\xb0\xa3\xd8b\x8b\x1d5\xf6\xde+\x8a\x1a[\xd4h\x12\x8d\x1a5v\x8d\xfd\x9d\xa8I\x8cQ\xb1w\xa3\xc6\xae1\xd6X\xa2&j\xb0b\x01\x15\xb0\x00\x02\x02\xd2\x9f\xdf\x1f\xfe\x98/\x0b\xdb\xb0\xad\xc9\xe7~]\x17\x97\xee\x9c)g\xce\x9c9s\

...

'fmt': 'image/png', 'fwidth': 1706, 'fheight': 665, '_blink_flag': False, '_qpix_orig': <PyQt5.QtGui.QPixmap object at 0x7f2b6c0456d0>, '_qpix_scaled': <PyQt5.QtGui.QPixmap object at 0x7f2b6c063740>}

Any thoughts on how I could release these references? After closing the console, or removing all plots, I’d expect them to be freed from memory. For example, gc.get_referrers(gc.get_objects()[-95]) (where gc.get_objects()[-95] is a png from matplotlib) lists <spyder.plugins.plots.widgets.figurebrowser.FigureCanvas object at 0x7fe79495cdc0>