'Run All Tests' button does not invoke Auto Save

Environment data

• VS Code version: 1.38.1
• Extension version (available under the Extensions sidebar): 2019.10.37964-dev
• OS and version: MS Windows 10 Pro 1903 18362.356 64bits
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.7 64-bit
• Type of virtual environment used (N/A | venv | virtualenv | conda | …): venv
• Relevant/affected Python packages and their versions: pytest 5.1.3
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): Jedi
• Settings has Auto Save enabled with onFocusChange option selected
• pytest was selected as test framework

Expected behaviour

When a file is dirty, clicking on the button ‘Run All Tests’ or ‘Run Failed Tests’ in the Test Explorer saves the dirty file before the tests are run. Tests are run with the updated code. Similar for the ‘Discover Tests’ button.

Actual behaviour

When a file is dirty, clicking on the button ‘Run All Tests’ or ‘Run Failed Tests’ in the Test Explorer does not save the dirty file before the tests are run. Tests are run with the outdated code. Similar for the ‘Discover Tests’ button.

The dirty file is saved when clicking clicking the ‘Show Test Output’ or ‘Collapse All’ buttons.

Steps to reproduce:

1. IMHO obvious from above description, but I’ll provide them if you cannot reproduce.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~\venvs\rentomatic-env\Scripts\python.exe -m pytest --junitxml=C:\Users\roger\AppData\Local\Temp\tmp-95569HLLaMJLU5V3.xml -s
cwd: c:\Users\roger\projects\py\rentomatic

Normally, the linter is invoked when focus shifts:

> ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
cwd: c:\Users\roger\projects\py\rentomatic
> ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
cwd: c:\Users\roger\projects\py\rentomatic
##########Linting Output - pylint##########

--------------------------------------------------------------------

Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

[Extension Host] Info Python Extension: 2019-09-24 22:15:05: Cached data exists ActivatedEnvironmentVariables, c:\Users\roger\projects\py\rentomatic
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: getActivatedEnvironmentVariables, Class name = _, completed in 1ms, Arg 1: <Uri:c:\Users\roger\projects\py\rentomatic>, Arg 2: undefined, Arg 3: undefined
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: > ~\venvs\rentomatic-env\Scripts\python.exe -m pytest --junitxml=C:\Users\roger\AppData\Local\Temp\tmp-95562XVhzmAhK5Nb.xml -s
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:05: cwd: c:\Users\roger\projects\py\rentomatic

And only after focus shifts:

[Extension Host] Info Python Extension: 2019-09-24 22:15:26: Cached data exists getEnvironmentVariables, c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: > ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: cwd: c:\Users\roger\projects\py\rentomatic
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: > ~\venvs\rentomatic-env\Scripts\python.exe -m pylint --disable=all --enable=F,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,unused-wildcard-import,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode,E0001,E0011,E0012,E0100,E0101,E0102,E0103,E0104,E0105,E0107,E0108,E0110,E0111,E0112,E0113,E0114,E0115,E0116,E0117,E0118,E0202,E0203,E0211,E0213,E0236,E0237,E0238,E0239,E0240,E0241,E0301,E0302,E0303,E0401,E0402,E0601,E0602,E0603,E0604,E0611,E0632,E0633,E0701,E0702,E0703,E0704,E0710,E0711,E0712,E1003,E1101,E1102,E1111,E1120,E1121,E1123,E1124,E1125,E1126,E1127,E1128,E1129,E1130,E1131,E1132,E1133,E1134,E1135,E1136,E1137,E1138,E1139,E1200,E1201,E1205,E1206,E1300,E1301,E1302,E1303,E1304,E1305,E1306,E1310,E1700,E1701 --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text c:\Users\roger\projects\py\rentomatic\tests\domain\test_room.py
console.ts:137 [Extension Host] Info Python Extension: 2019-09-24 22:15:26: cwd: c:\Users\roger\projects\py\rentomatic

[rogererens]

Hi Don,

thanks for picking up this one. I’m not sure I fully understand the (second part of the) design decision: some buttons in Test Explorer lead to saving the file, other buttons in Test Explorer don’t. As I see it, it is the responsibility of the Editor to save a file when it looses focus (given my chosen option onFocusChange), be that either to an extenstion’s panel, the Terminal panel, or even a browser or MS Word. Anyway, thanks for this awesome piece of work by you and your team!

[CBerndt-Th]

As far as I can tell the issue is still in effect. I’ve found this ticket after looking how to save the current test file when running a single test from the test mark in the file, although that probably would be a separate issue.

I’m unsure what the [needs proposal] label means and couldn’t find anything in the wiki, but here is are my two cents: I’d propose to save all files on test execution in general with a settings flag to enable this behavior. The default should be false. That solves the issue initially described, my use case and is compatible with previous experience.