[BUG] Spyder debug-mode returns "*** NameError: name 'labels' is not defined" when existing list object 'labels' is used in a comprehension

Problem Description / Task Background

I have a function that I use to generate complex plots of different data sets with consistent formatting. There are multiple lines on each plot, but I only want a subset of those lines included in the legend. To get rid of the unwanted labels, I start by grabbing the handles and labels with get_legend_handles_labels() from matplotlib. Once I have the list of labels, I identify the index (the idx array) and use a comprehension build a list of the labels I want to keep. The comprehension is as follows:

la = [labels[index] for index in idx]

In debug mode, if I break on this line and then enter it into the iPython console I get the following error:

*** NameError: name 'labels' is not defined

NOTES:

1. This error does not appear if I run the code in standard run mode. The plot (with the truncated legend) is generated as expected.
2. ‘labels’ does exist at this point in the code - if I type ‘labels’ into the console it returns the contents of labels.
3. This error does does not appear in debug mode if I do not place the plot code in a separate function. That is, the break point is in the same file that I started debugging, instead of being in a function called by the file I’m debugging.

What steps reproduce the problem?

To demonstrate this problem (in a highly simplified manner), I wrote three .py files:

1. Spyder_Error_driver.py: Loads and calls the Spyder_Error_Demo.py function. The code is:

import Spyder_Error_Demo as SED
labels_keep = SED.Spyder_Error_Demo()
print(labels_keep)

1. Spyder_Error_Demo.py: Makes a list, defines indices to keep, and uses a comprehension to extract the ‘keep’ labels. The code is:

import numpy as np
def Spyder_Error_Demo():
    labels = ['keep','other','keep2','other']
    idx = np.array([0,2])
    labels_keep = [labels[index] for index in idx]
    return labels_keep

1. Spyder_Error_Demo_script.py: Performs the same operations as the other two files, but without calling a function. The code is:

import numpy as np
labels = ['keep','other','keep2','other']
idx = np.array([0,2])
labels_keep = [labels[index] for index in idx]
print(labels_keep)

To show that the code works as intended, paste the code for the three files into .py files and save with the names provided above. Then run:

1. Spyder_Error_Demo.py
2. Spyder_Error_Demo_script.py
3. Both will print: ['keep', 'keep2']

To reproduce the problem:

1. Place a breakpoint at line 5 of Spyder_Error_Demo.py
2. Run Spyder_Error_driver.py in debug mode
3. Continue to the breakpoint on line 5
4. Type “labels” into the iPython console. This will show the contents of ‘labels’
5. Paste labels_keep = [labels[index] for index in idx] into the iPython console. This results in an error: *** NameError: name 'labels' is not defined.

To show that this only seems to appear when debugging a function:

1. Place a breakpoint at line 4 of Spyder_Error_Demo_script.py
2. Run Spyder_Error_Demo_script.py in debug mode
3. Continue to the breakpoint on line 4
4. Paste labels_keep = [labels[index] for index in idx] into the iPython console. This does not result in an error and the ‘labels_keep’ variable is created as expected.

Versions

• Anaconda: 1.8.7
• Spyder version: 3.2.8
• Python version: 3.6
• Qt version: 5.9.6
• PyQt version: 5.9.2
• Operating System name/versions: Windows 10 & Windows 7 & Mint VM on Windows 10

Dependencies

IPython >=4.0    :  6.4.0 (OK)
cython >=0.21    :  0.28.3 (OK)
jedi >=0.9.0     :  0.12.0 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
numpy >=1.7      :  1.14.5 (OK)
pandas >=0.13.1  :  0.23.1 (OK)
pycodestyle >=2.3:  2.4.0 (OK)
pyflakes >=0.6.0 :  2.0.0 (OK)
pygments >=2.0   :  2.2.0 (OK)
pylint >=0.25    :  1.9.2 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
rope >=0.9.4     :  0.10.7 (OK)
sphinx >=0.6.6   :  1.7.5 (OK)
sympy >=0.7.3    :  1.1.1 (OK)

[bcolsen]

@jeremymatt

It is unfortunate that interact can’t be exited out of though.

The fix is on the way: https://github.com/jupyter/qtconsole/pull/290

This is just to satisfy my own curiosity: is the reason that the error does not occur when debugging a script because the variables in that script are treated as global (this seems odd to me)?

Yes python scripts are almost highest level of execution second only to python’s own built-ins. Everything, other than built-ins, that a python script uses is defined in the script either through imports or directly in the script it’s self.

Python’s name space hierarchy looks like this:

Local -> Enclosed -> Global -> Built-in,

where the arrows should denote the direction of the namespace-hierarchy search order.

• Local can be inside a function or class method, for example.
• Enclosed can be its enclosing function, e.g., if a function is wrapped inside another function.
• Global refers to the uppermost level of the executing script itself, and
• Built-in are special names that Python reserves for itself.

quote from: https://sebastianraschka.com/Articles/2014_python_scope_and_namespaces.html

To explain the original problem here, list comprehensions are functions in python3 with their own namespace. A list comprehension defined in the debugger is probably not being “enclosed” by the function so it can’t see the function’s local namespace. The interact command also just globals the locals to make the environment more script like.

[bcolsen]

@jeremymatt Thank you for the in-depth bug report! I know how much work that can be.

This an interesting issue and it seems to be related to this python bug https://bugs.python.org/issue21161

This is solved by using the interact command at the pdb prompt:

This seems like the official way to work with this type of problem, so I’m not sure if there is much to be done in Spyder. Unfortunately, there maybe bug in Spyder because I can’t exit the pdb interactive mode without restarting the console, but I’ll submit that separately.

There is also a work around where you can run globals().update(locals()) in pdb before running labels_keep = [labels[index] for index in idx]. That will also work, by it might cause some strange things to happen in more complicated code.