Failed Pester 5 tests are not shown in Problems

Issue Description

When using Pester 5.0.3 to execute a test task, the failed tests are not shown in Problems-view like they used to do with Pester 3.4. This seems to be caused by the new output-format in Pester 5 which doesn’t fit with the $pester-problemMatcher regex.

Replacing the current matcher ^\s*(?:\[-\]\s+)(.*?)(?:\s+\d+\.?\d*\s*m?s)\s*$ with ^\s*(?:\[-\]\s+)(.*?)(?:\s+\d+\.?\d*\s*m?s)(?:\s+\(.*?\))?\s*$ in a regex online tester seems to detect both Pester 3.4 and Pester 5 output, but I had problems making it work by editing directly in package.json (remember to escape \), so might need something more.

Proof-of-concept(Click to Expand)

demo.tests.ps1

Describe "VSCode Problem demo" {
    Context "MyContext" {
        if((Get-Module Pester).Version -ge "5.0") {
            It "fails Pester5" {
                1 | Should -Be 2
            }
        } else {
            It "fails Pester" {
                1 | Should Be 2
            }
        }
    }
}

tasks.json

{
    "version": "2.0.0",
    "windows": {
        "options": {
            "shell": {
                "executable": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
                "args": [
                    "-NoProfile",
                    "-ExecutionPolicy",
                    "Bypass",
                    "-Command"
                ]
            }
        }
    },
    "tasks": [
        {
            "label": "Test",
            "type": "shell",
            "command": "Import-Module 'C:\\Program Files\\WindowsPowerShell\\Modules\\Pester\\3.4.0\\Pester.psd1'; Invoke-Pester -PesterOption @{IncludeVSCodeMarker=$true}",
            "group": {
                "kind": "test",
                "isDefault": true
            },
            "problemMatcher": [
                "$pester"
            ]
        },
        {
            "label": "TestPester5",
            "type": "shell",
            "command": "Import-Module 'C:\\Program Files\\WindowsPowerShell\\Modules\\Pester\\5.0.3\\Pester.psd1'; Invoke-Pester -Configuration ([PesterConfiguration]@{Output=@{Verbosity = 'Detailed'}; Debug = @{ ShowNavigationMarkers = $true } })",
            "group": {
                "kind": "test",
                "isDefault": true
            },
            "problemMatcher": [
                "$pester"
            ]
        }
    ]
}

Output Pester 3.4 (shown in Problems-tab):

Describing VSCode Problem demo
   Context MyContext
    [-] fails Pester 398ms
      at <ScriptBlock>, C:\Sandbox\demo.tests.ps1: line 9
      Expected: {2}
      But was:  {1}
      9:                 1 | Should Be 2
      at <ScriptBlock>, C:\Sandbox\demo.tests.ps1: line 9
Tests completed in 398ms
Passed: 0 Failed: 1 Skipped: 0 Pending: 0 Inconclusive: 0

Output Pester 5.0.3 (not shown in Problems-tab):

Starting discovery in 1 files.
Discovering in demo.tests.ps1.
Found 1 tests. 95ms
Discovery finished in 192ms.

Running tests from 'demo.tests.ps1'
Describing VSCode Problem demo, C:\Sandbox\demo.tests.ps1:1
 Context MyContext, C:\Sandbox\demo.tests.ps1:2
   [-] fails Pester5, C:\Sandbox\demo.tests.ps1:4 124ms (94ms|30ms)
    Expected 2, but got 1.
    at 1 | Should -Be 2, C:\Sandbox\demo.tests.ps1:5
    at <ScriptBlock>, C:\Sandbox\demo.tests.ps1:5
Tests completed in 727ms
Tests Passed: 0, Failed: 1, Skipped: 0 NotRun: 0

Attached Logs

Follow the instructions in the README about capturing and sending logs.

Environment Information

Visual Studio Code

Name	Version
Operating System	Windows_NT x64 10.0.19041
VSCode	1.48.2
PowerShell Extension Version	2020.6.0

PowerShell Information

Name	Value
PSVersion	7.0.3
PSEdition	Core
GitCommitId	7.0.3
OS	Microsoft Windows 10.0.19041
Platform	Win32NT
PSCompatibleVersions	1.0 2.0 3.0 4.0 5.0 5.1.10032.0 6.0.0 6.1.0 6.2.0 7.0.3
PSRemotingProtocolVersion	2.3
SerializationVersion	1.1.0.1
WSManStackVersion	3.0

Visual Studio Code Extensions

Visual Studio Code Extensions(Click to Expand)

Extension	Author	Version
azure-account	ms-vscode	0.9.2
azure-pipelines	ms-azure-devops	1.174.2
gc-excelviewer	GrapeCity	3.0.40
githistory	donjayamanne	0.6.9
gitlens	eamodio	10.2.2
hexeditor	ms-vscode	1.2.1
ilspy-vscode	icsharpcode	0.9.0
material-icon-theme	PKief	4.2.0
material-theme	zhuangtongfa	3.8.7
open-in-browser	techer	2.0.0
powershell	ms-vscode	2020.6.0
rainbow-csv	mechatroner	1.7.1
reg	ionutvmi	1.0.1
remote-containers	ms-vscode-remote	0.134.1
remote-ssh	ms-vscode-remote	0.51.0
remote-ssh-edit	ms-vscode-remote	0.51.0
remote-wsl	ms-vscode-remote	0.44.5
vscode-azurefunctions	ms-azuretools	0.24.0
vscode-azurestorage	ms-azuretools	0.9.0
vscode-docker	ms-azuretools	1.5.0
vscode-remote-extensionpack	ms-vscode-remote	0.20.0
vscode-test-explorer	hbenl	2.19.1
vsliveshare	ms-vsliveshare	1.0.2731
xml	DotJoshJohnson	2.5.1

[SydneyhSmith]

Thanks @nohwnd for the update, on the vscode side looks like the change would happen here https://github.com/powershell/vscode-powershell/blob/master/package.json#L456-L474 I will go ahead and mark this issue as up for grabs

[nohwnd]

Oh, nevermind, I see it’s a 5.x update problem. BeforeAll can’t be in Context anymore.

It can, it’s just that your BeforeAll is in Context that has no It, so BeforeAll won’t run because it would not setup any test.