Spyder does not free up memory after closing windows with datasets in the Variable explorer

Description

What steps will reproduce the problem?

0.~Optionally, display memory usage (Preferences --> General --> Advanced Settings --> Check the box named ‘Show memory usage’)

1. Read a large data set. For ex.: .csv 300 MB (the link may not be assessible after November, 6th, 2016) using the following code

import pandas as pd raw = pd.read_csv('transactions.csv')

2.~Once you have read it into your RAM, try to glimpse at it by double clicking in the Variable Explorer. You will see a window that displays your data set. 3. Close this window 4. Repeat Step 2 until your RAM is full.

What is the expected output? What do you see instead?

1. It is reasonable to show only a couple of hundreds of rows and when a user wants more – load more.
2. Free up RAM after Step 3.

Version and main components

• Spyder Version: 3.0.0
• Python Version: 3.5.2
• Qt Versions: 4.8.7, PyQt4 (API v2) 4.11.4 on Linux

Dependencies

pyflakes >=0.6.0 :  1.3.0 (OK)
pep8 >=0.6       :  1.7.0 (OK)
pygments >=2.0   :  2.1.3 (OK)
qtconsole >=4.2.0:  4.2.1 (OK)
nbconvert >=4.0  :  4.2.0 (OK)
pandas >=0.13.1  :  0.17.1 (OK)
numpy >=1.7      :  1.11.0 (OK)
sphinx >=0.6.6   :  1.4.6 (OK)
rope >=0.9.4     :  0.9.4-1 (OK)
jedi >=0.8.1     :  0.9.0 (OK)
psutil >=0.3     :  4.3.1 (OK)
matplotlib >=1.0 :  1.5.1 (OK)
sympy >=0.7.3    :  None (NOK)
pylint >=0.25    :  1.6.4 (OK)

[ccordoba12]

We need to garbage-collect values we grab from the kernel after users close our viewers. I’ll try to do that for 3.0.2 😃

[ccordoba12]

It looks like all data is copied when you open the dataframe editor

That’s correct, data is serialized in the kernel and sent to Spyder so we can show it with our different editors.

I get the impression that only a small amount of data is supposed to be copied, not the whole dataframe

Nop, what we do is to show data in chunks in the dataframe editor, but at any moment we have access to the full dataframe. I don’t know how we could do it otherwise 😃

an overview would be helpful

Ok, so these are the steps we follow to show a value in one of our editors:

1. We ask the console for the value of a variable:

   https://github.com/spyder-ide/spyder/blob/master/spyder/widgets/variableexplorer/namespacebrowser.py#L333

2. This sends a petition to the kernel:

   https://github.com/spyder-ide/spyder/blob/master/spyder/widgets/ipythonconsole/namespacebrowser.py#L72

3. The kernel serializes the value it was asked for, with the help of publish_data:

   https://github.com/spyder-ide/spyder/blob/master/spyder/utils/ipython/spyder_kernel.py#L132

4. That value is received by Spyder in _handle_data_message, deserialized and saved in _kernel_value:

   https://github.com/spyder-ide/spyder/blob/master/spyder/widgets/ipythonconsole/namespacebrowser.py#L143

5. Finally, this value is passed to our editors:

   https://github.com/spyder-ide/spyder/blob/master/spyder/widgets/variableexplorer/collectionseditor.py#L365

I know this is very complex, but we have to do all this because the kernel runs in an external process (which can be local or remote, i.e. in a different server 😃