Jupyter: Some lists and pd.DataFrames not opening in Data Viewer

Environment data

• VS Code version: 1.53.2 (user setup)
• Jupyter Extension version: v2021.2.576440691
• Python Extension version (available under the Extensions sidebar): v2021.2.582707922
• OS (Windows | Mac | Linux distro) and version: Windows_NT x64 10.0.19042
• Python and/or Anaconda version: Python 3.7.8 64-bit
• Type of virtual environment used: conda
• Jupyter server running: Local

Expected behaviour

Opening all lists and pd.DataFrames in Data Viewer should display their contents.

Actual behaviour

Some lists and DataFrames are not displayed in Data Viewer.

Steps to reproduce:

1. Make a list of files (paths to files)

import pathlib

# make a long path
path = pathlib.Path.cwd()
new_path = path / "some_analysis_folder_with_data" / "some_output_folder"
pathlib.Path(new_path).mkdir(parents=True, exist_ok=True)

# make long files in the folder
file_name_stem = "some_file_name_foooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo"
file_suffix = ".gz"

for i in range(1,8):
    new_file = new_path / f"{file_name_stem}_{i}{file_suffix}"
    pathlib.Path(new_file).touch()

new_files = [file for file in new_path.iterdir()]
# type(new_files[0]) # pathlib.WindowsPath
new_files # this list doesn't open in Data Viewer

1. Make a list of “fake” files, i.e., strings, not pathlib.WindowsPath objects

# make a new list (fake_files_list) similar to new_files, but with strings, not pathlib.WindowsPath objects
## make sure the elements in fake_files_list are as long as those in new_files

fake_folder_name = str(new_file.parent)
letters_for_fake_file = string.ascii_lowercase[:len(new_files)]
new_file_stem_len = len(str(new_file.stem[:-2]))

fake_files_list = []

for i in range(len(new_files)):
    fake_files_list.append(f"{fake_folder_name}\\{letters_for_fake_file[i]*new_file_stem_len}_{i}{file_suffix}")

for i in range(len(new_files)):
    assert len(fake_files_list[i]) == len(str(new_files[i])), f"ERROR: Lenghts are different for element {i}"

fake_files_list

1. fake_files_list opens in Data Viewer, but new_files doesn’t. I have a similar issue with some DataFrames. The size of the Dataframe does not seem to matter.

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

> ~\Anaconda3\envs\mobs_abm\python.exe -c "import pandas;print(pandas.__version__)"
Error 2021-02-19 22:09:28: Failure during variable extraction:
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-3-8fc409734a9b> in <module>()
----> 1 print(_VSCODE_getDataFrameRows(new_files, 0, 7))

<ipython-input-3-93cda1340104> in _VSCODE_getDataFrameRows(df, start, end)
    136     except:
    137         pass
--> 138     return _VSCODE_pd_json.to_json(None, rows, orient="table", date_format="iso")
    139 
    140 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in to_json(path_or_buf, obj, orient, date_format, double_precision, force_ascii, date_unit, default_handler, lines, compression, index, indent)
     82         default_handler=default_handler,
     83         index=index,
---> 84         indent=indent,
     85     ).write()
     86 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in write(self)
    142             self.date_format == "iso",
    143             self.default_handler,
--> 144             self.indent,
    145         )
    146 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    339             iso_dates,
    340             default_handler,
--> 341             indent,
    342         )
    343 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    242             iso_dates,
    243             default_handler,
--> 244             indent,
    245         )
    246 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    164             iso_dates=iso_dates,
    165             default_handler=default_handler,
--> 166             indent=indent,
    167         )
    168 

OverflowError: Maximum recursion level reached
Error 2021-02-19 22:09:28: [Error: Failure during variable extraction:
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-3-8fc409734a9b> in <module>()
----> 1 print(_VSCODE_getDataFrameRows(new_files, 0, 7))

<ipython-input-3-93cda1340104> in _VSCODE_getDataFrameRows(df, start, end)
    136     except:
    137         pass
--> 138     return _VSCODE_pd_json.to_json(None, rows, orient="table", date_format="iso")
    139 
    140 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in to_json(path_or_buf, obj, orient, date_format, double_precision, force_ascii, date_unit, default_handler, lines, compression, index, indent)
     82         default_handler=default_handler,
     83         index=index,
---> 84         indent=indent,
     85     ).write()
     86 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in write(self)
    142             self.date_format == "iso",
    143             self.default_handler,
--> 144             self.indent,
    145         )
    146 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    339             iso_dates,
    340             default_handler,
--> 341             indent,
    342         )
    343 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    242             iso_dates,
    243             default_handler,
--> 244             indent,
    245         )
    246 

C:\Users\Christine\Anaconda3\envs\mobs_abm\lib\site-packages\pandas\io\json\_json.py in _write(self, obj, orient, double_precision, ensure_ascii, date_unit, iso_dates, default_handler, indent)
    164             iso_dates=iso_dates,
    165             default_handler=default_handler,
--> 166             indent=indent,
    167         )
    168 

OverflowError: Maximum recursion level reached
	at k.extractJupyterResultText (c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:572275)
	at k.deserializeJupyterResult (c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:572394)
	at k.getDataFrameRows (c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:570203)
	at async r.getRows (c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:210853)
	at async c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:206244
	at async C.wrapRequest (c:\Users\Christine\.vscode\extensions\ms-toolsai.jupyter-2021.2.576440691\out\client\extension.js:49:206473)]

Log from Extension Host.

[2021-02-19 22:02:01.803] [exthost] [warning] [Decorations] CAPPING events from decorations provider vscode.git 4157
[2021-02-19 22:06:00.763] [exthost] [info] ExtensionService#_doActivateExtension vscode.typescript-language-features {"startup":false,"extensionId":{"value":"vscode.typescript-language-features","_lower":"vscode.typescript-language-features"},"activationEvent":"onLanguage:jsonc"}
[2021-02-19 22:06:00.764] [exthost] [info] ExtensionService#loadCommonJSModule file:///c:/Users/Christine/AppData/Local/Programs/Microsoft VS Code/resources/app/extensions/typescript-language-features/dist/extension
[2021-02-19 22:06:06.486] [exthost] [warning] [Decorations] CAPPING events from decorations provider vscode.git 4157

[DavidKutu]

Thanks for reporting it @maciejkos, and thanks for your detailed repro steps. I can also get it using regular python. I’ll bring this up in our next meeting.

[joyceerhl]

Update: we fixed the problem with transposed DataFrames in the April release. The original problem with objects not being displayed would be resolved by https://github.com/microsoft/vscode-jupyter/issues/1138 so closing as a duplicate of that. Please feel free to upvote that issue instead.