Extension loads forever when there are a lot of conda environments (remote SSH)

Environment data

• VS Code version: 1.43.2 (user setup)
• Extension version (available under the Extensions sidebar):
  • local:
    • Remote development 0.20.0
    • Remote SSH 0.51.0
• OS and version: Windows 10 Enterprise 1709 16299.1686
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.7 (from conda default channel)
• Type of virtual environment used (N/A | venv | virtualenv | conda | …): conda
• Relevant/affected Python packages and their versions: Not relevant from my understanding
• Relevant/affected Python-related VS Code extensions and their versions: Python extension 2020.3.69010
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): Jedi (python.JediEnabled = True)
• Value of the python.languageServer setting: Microsoft

Expected behaviour

Editing python files is snappy.

Actual behaviour

Python extension loading… message runs indefinitely.

Steps to reproduce:

Open any folder or multi root workspace on a CentOS 7 machine and this behavior is shown.

Logs

Output of Remote - SSH

[12:48:20.332] Log Level: 1
[12:48:20.379] remote-ssh@0.51.0
[12:48:20.379] win32 x64
[12:48:20.382] SSH Resolver called for "ssh-remote+poisux803", attempt 1
[12:48:20.383] SSH Resolver called for host: poisux803
[12:48:20.383] Setting up SSH remote "poisux803"
[12:48:20.421] Using commit id "0ba0ca52957102ca3527cf479571617f0de6ed50" and quality "stable" for server
[12:48:20.423] Install and start server if needed
[12:48:20.430] Checking ssh with "ssh -V"
[12:48:24.138] > OpenSSH_7.9p1, OpenSSL 1.1.1a  20 Nov 2018
[12:48:24.141] Running script with connection command: ssh -T -D 62632 poisux803 bash
[12:48:24.146] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[12:48:25.386] "install" terminal received data: ""
[12:48:25.387] Got some output, clearing connection timeout
[12:48:27.655] "install" terminal received data: "e56d19b5f442: running[?25l
Acquiring lock on /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50
\ln /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50.target /import/poi/home/poimat04/.vscode-server/bin/0ba0ca5
2957102ca3527cf479571617f0de6ed50/vscode-remote-lock.poimat04.0ba0ca52957102ca3527cf479571617f0de6ed50
[?25h"
[12:48:27.721] "install" terminal received data: "Found existing installation at /import/poi/home/poimat04/.vscode-server/bin/0ba0ca52957102ca3527cf479571617f0de6ed50...[?25l
[?25h"
[12:48:27.740] "install" terminal received data: "Found running server...[?25l

*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
*

[?25h"
[12:48:27.776] "install" terminal received data: "Checking server status on port 37978 with wget[?25l
e56d19b5f442: start
sshAuthSock====
agentPort==37978==
osReleaseId==centos==
arch==x86_64==
webUiAccessToken====
tmpDir==/run/user/10029==
platform==linux==
e56d19b5f442: end
[?25h"
[12:48:27.776] Received install output: 
sshAuthSock====
agentPort==37978==
osReleaseId==centos==
arch==x86_64==
webUiAccessToken====
tmpDir==/run/user/10029==
platform==linux==

[12:48:27.777] Remote server is listening on port 37978
[12:48:27.777] Parsed server configuration: {"agentPort":37978,"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","tmpDir":"/run/user/10029","platform":"linux"}
[12:48:27.780] Starting forwarding server. localPort 62659 -> socksPort 62632 -> remotePort 37978
[12:48:27.781] Forwarding server listening on 62659
[12:48:27.781] Waiting for ssh tunnel to be ready
[12:48:27.784] Tunneled remote port 37978 to local port 62659
[12:48:27.784] Resolved "ssh-remote+poisux803" to "127.0.0.1:62659"
[12:48:27.786] [Forwarding server 62659] Got connection 0
[12:48:27.796] TELEMETRY: {"eventName":"resolver","properties":{"outcome":"success","osReleaseId":"centos","arch":"x86_64","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0"},"measures":{"resolveAttempts":1,"retries":1}}
[12:48:27.798] ------




[12:48:27.806] [Forwarding server 62659] Got connection 1
[12:48:27.807] [Forwarding server 62659] Got connection 2
[12:48:29.015] [Forwarding server 62659] Got connection 3
[12:48:29.017] [Forwarding server 62659] Got connection 4
[12:48:29.019] [Forwarding server 62659] Got connection 5

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

I can see several instances of those errors:

Error: Trying to add a disposable to a DisposableStore that has already been disposed of. The added object will be leaked!
    at l.add (lifecycle.ts:134)
    at e._register (lifecycle.ts:164)
    at e.addDisposable (terminalInstance.ts:343)
    at m._initInstanceListeners (terminalTab.ts:272)
    at new m (terminalTab.ts:246)
    at h._createInstance (instantiationService.ts:117)
    at h.createInstance (instantiationService.ts:82)
    at T._showBackgroundTerminal (terminalService.ts:597)
    at T.setActiveInstance (terminalService.ts:316)
    at p.$show (mainThreadTerminalService.ts:108)
    at g._doInvokeHandler (rpcProtocol.ts:398)
    at g._invokeHandler (rpcProtocol.ts:383)
    at g._receiveRequest (rpcProtocol.ts:299)
    at g._receiveOneMessage (rpcProtocol.ts:235)
    at rpcProtocol.ts:101
    at c.fire (event.ts:587)
    at v.fire (ipc.net.ts:453)
    at t.PersistentProtocol._receiveMessage (ipc.net.ts:733)
    at ipc.net.ts:592
    at c.fire (event.ts:587)
    at g.acceptChunk (ipc.net.ts:239)
    at ipc.net.ts:200
    at Socket.t (ipc.net.ts:28)
    at Socket.emit (events.js:203)
    at addChunk (_stream_readable.js:295)
    at readableAddChunk (_stream_readable.js:276)
    at Socket.Readable.push (_stream_readable.js:210)
    at TCP.onStreamRead (internal/stream_base_commons.js:166)
add @ lifecycle.ts:134
_register @ lifecycle.ts:164
addDisposable @ terminalInstance.ts:343
_initInstanceListeners @ terminalTab.ts:272
m @ terminalTab.ts:246
_createInstance @ instantiationService.ts:117
createInstance @ instantiationService.ts:82
_showBackgroundTerminal @ terminalService.ts:597
setActiveInstance @ terminalService.ts:316
$show @ mainThreadTerminalService.ts:108
_doInvokeHandler @ rpcProtocol.ts:398
_invokeHandler @ rpcProtocol.ts:383
_receiveRequest @ rpcProtocol.ts:299
_receiveOneMessage @ rpcProtocol.ts:235
(anonymous) @ rpcProtocol.ts:101
fire @ event.ts:587
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:587
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:203
addChunk @ _stream_readable.js:295
readableAddChunk @ _stream_readable.js:276
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

I can see this also:

console.ts:137 [Extension Host] Error Python Extension: 2020-03-30 12:53:07: stderr jediProxy Error (stderr) completion.py:589: DeprecationWarning: Deprecated since version 0.16.0. Use Script(...).get_names instead.  all_scopes=True,
t.log @ console.ts:137
$logExtensionHostMessage @ mainThreadConsole.ts:39
_doInvokeHandler @ rpcProtocol.ts:398
_invokeHandler @ rpcProtocol.ts:383
_receiveRequest @ rpcProtocol.ts:299
_receiveOneMessage @ rpcProtocol.ts:226
(anonymous) @ rpcProtocol.ts:101
fire @ event.ts:587
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:587
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:203
addChunk @ _stream_readable.js:295
readableAddChunk @ _stream_readable.js:276
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

I see also a huge amount of messages saying that the extension is searching for interpreters

[Extension Host] Info Python Extension: 2020-03-30 12:52:37: Get Interpreters in CacheableLocatorService, Class name = m, completed in 24ms, has a truthy return value, Arg 1: <Uri:/import/projects/cfd/Matthieu/Workspace/oplib/oplib/taskmanager/OPRunManager.py>, Return Value: []

My sysadmin on the server host told me that when doing a trace on the SSH traffic there was 200k messages in a few minutes and my VPN client says that there are GB of data exchanged which surprises us.

[matmel]

Does VSCode load properly When the the python extension is disabled.

Yes as fast as usual. And I can edit easily files afterwards. Whereas when I am trying to even save a python file when the extension is loading indefinitely, the actual save can happen minutes later.

[matmel]

• When you say that you use conda, you mean on the remote machine?

Yes. I also use it on the local machine but I think it is irrelevant here.

• If you open a non-Python file on your remote (so the Python extension doesn’t get activated), do you still get an important volume of messages?

No the volume of messages is much much reduced in that case.

• Can you reproduce this issue when creating a folder and adding a python file to it, instead of opening existing projects?

Yes, same problem in a single empty folder workspace with a one liner python script:

$ cat foo.py
from pathlib import Path

• What is the output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)?

Empty in the case of my big workspace. In the small one of the previous test:

User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'ShowExtensionSurveyPrompt - enabled'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'Reload - experiment'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'AA_testing - control'
User belongs to experiment group 'WebHostNotebook - control'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> /import/common/conda/bin/conda info --json
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"

and it is frozen here.

I can add that the amount of network messages seems similar to the big workspace. Just to give you an idea, here is the output of a tcpdump filtered with my laptop ip and ssh traffic only (I have no other ssh traffic on this host except for vscode)

I also tried to downgrade the python extension on the host 2020.1.xxxxxx, because this problem seems new. Unfortunately it doesn’t help. Maybe it is due to the recent upgrades of the remote ssh extension or because we are soliciting the VPN more in those troubled times?