DotNetCoreCLI dotnet test with coverage not working properly

Required Information

Question, Bug, or Feature?
Type: Bug

Enter Task Name: DotNetCoreCLIV2

Environment

• Server - Azure Pipelines
• Agent - Hosted - windows-2022

Issue Description

YAML:

  - task: DotNetCoreCLI@2
    displayName: dotnet test
    inputs:
      command: test
      projects: >-
        **\*UnitTests.csproj

        !**\obj\**
      arguments: --configuration $(BuildConfiguration) -p:CollectCoverage=true -p:CoverletOutput=$(Build.SourcesDirectory)\TestResults\XUnit\Coverage\ -p:CoverletOutputFormat=cobertura%2copencover "--collect:\"XPlat Code Coverage\""

Yesterday (2022/11/23) it was working properly. Today (2022/11/24) the issue started. It does not fail on the referred step, but it does not generate the expected coverage output and a depending step fails to find the expected file. The same call works in windows command line, generating the expected file. Note that nothing relevant in the repository or in the pipeline changed. We noticed the same problem is happening in another independent project/pipeline.

Task logs

Yesterday, the logs were like this:

Starting test execution, please wait… A total of 1 test files matched the specified pattern. Results File: D:\a_temp\VssAdministrator_fv-az295-991_2022-11-23_12_44_42.trx

Attachments: D:\a_temp\962ef98d-1373-44fd-a7d7-f97c7cb2bc12\coverage.cobertura.xml Passed! - Failed: 0, Passed: 10, Skipped: 0, Total: 10, Duration: 128 ms - ProjectNamePlaceholder.UnitTests.dll (net6.0)

Calculating coverage result… Generating report ‘D:\a\1\s\TestResults\XUnit\Coverage\coverage.cobertura.xml’ Generating report ‘D:\a\1\s\TestResults\XUnit\Coverage\coverage.opencover.xml’

±---------------------------------------±------±-------±-------+ | Module | Line | Branch | Method | ±---------------------------------------±------±-------±-------+ | ProjectNamePlaceholder.FunctionApp | 64.6% | 77.77% | 64.28% | ±---------------------------------------±------±-------±-------+ | ProjectNamePlaceholder.Data | 0.52% | 0% | 28.4% | ±---------------------------------------±------±-------±-------+

±--------±-------±-------±-------+ | | Line | Branch | Method | ±--------±-------±-------±-------+ | Total | 0.64% | 12.72% | 29.35% | ±--------±-------±-------±-------+ | Average | 32.55% | 38.88% | 46.34% | ±--------±-------±-------±-------+

(Afterwards another project is tested)

Today, the logs are like this:

Starting test execution, please wait… A total of 1 test files matched the specified pattern. Results File: D:\a_temp\VssAdministrator_WIN-LMUMTU2TIJQ_2022-11-24_12_49_47.trx

Passed! - Failed: 0, Passed: 10, Skipped: 0, Total: 10, Duration: 218 ms - ProjectNamePlaceholder.UnitTests.dll (net6.0)

Attachments: D:\a_temp\83cd84d5-1c62-4383-85d6-a01ba51e9fc9\coverage.cobertura.xml

(Afterwards another project is tested)

If we run in windows command line the command dotnet test -p:CollectCoverage=true -p:CoverletOutput=.\TestResults\XUnit\Coverage\ -p:CoverletOutputFormat=cobertura%2copencover "--collect:\"XPlat Code Coverage\"", we get the report and the logs are the following:

Determining projects to restore… All projects are up-to-date for restore. ProjectNamePlaceholder.Data -> c:(…)\src\ProjectNamePlaceholder.Data\bin\Debug\net6.0\ProjectNamePlaceholder.Data.dll ProjectNamePlaceholder.FunctionApp -> c:\Projects(…)\src\ProjectNamePlaceholder.FunctionApp\bi n\Debug\net6.0\ProjectNamePlaceholder.FunctionApp.dll ProjectNamePlaceholder.UnitTests -> c:(…)\src\ProjectNamePlaceholder.UnitTests\bin
Debug\net6.0\ProjectNamePlaceholder.UnitTests.dll Test run for c:(…)\src\ProjectNamePlaceholder.UnitTests\bin\Debug\net6.0\ProjectNamePlaceholder.UnitTests.dll (.NETCoreApp,Version=v6.0) Microsoft ® Test Execution Command Line Tool Version 17.2.0 (x64) Copyright © Microsoft Corporation. All rights reserved.

Starting test execution, please wait… A total of 1 test files matched the specified pattern.

Attachments: c:(…)\src\ProjectNamePlaceholder.UnitTests\TestResults\b70894de-7cd8-4c2c-adb0-080d6dc6e5cc\coverage.cobertura.xml Passed! - Failed: 0, Passed: 10, Skipped: 0, Total: 10, Duration: 185 ms - ProjectNamePlaceholder.UnitTests.dll (net6.0)

Calculating coverage result… Generating report ‘.\TestResults\XUnit\Coverage\coverage.cobertura.xml’ Generating report ‘.\TestResults\XUnit\Coverage\coverage.opencover.xml’

±---------------------------------------±-------±-------±-------+ | Module | Line | Branch | Method | ±---------------------------------------±-------±-------±-------+ | ProjectNamePlaceholder.FunctionApp | 64.78% | 77.77% | 64.28% | ±---------------------------------------±-------±-------±-------+ | ProjectNamePlaceholder.Data | 0.58% | 0% | 28.68% | ±---------------------------------------±-------±-------±-------+

±--------±-------±-------±-------+ | | Line | Branch | Method | ±--------±-------±-------±-------+ | Total | 0.73% | 12.72% | 29.63% | ±--------±-------±-------±-------+ | Average | 32.68% | 38.88% | 46.48% | ±--------±-------±-------±-------+

Error logs

2022-11-24T12:50:40: The report file pattern ‘D:\a\1\s\TestResults*\Coverage\coverage.opencover.xml’ found no matching files. 2022-11-24T12:50:40: No report files specified. ##[error]The process ‘C:\Program Files\dotnet\dotnet.exe’ failed with exit code 1

Please let me know if this is not the right place to raise this issue. Thank you!

[ablersch]

I had the same issue. Take a look at https://stackoverflow.com/questions/74386402/coverlet-does-not-create-report-in-azure-devops.

[joaocpribeiro]

Thank you, @kastrodev. I instead reverted the publishTestResults: false and started looking at opencover file in temp directory: $(Agent.TempDirectory)/*/coverage.opencover.xml

So, summarizing:

1. Replace the parameters of dotnet test: Before: -p:CollectCoverage=true -p:CoverletOutput=$(Build.SourcesDirectory)/TestResults/XUnit/Coverage/ -p:CoverletOutputFormat=cobertura%2copencover After: -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencovercobertura%2copencover
2. Search for coverage files in temp directory instead. In my case, I am using the task reportgenerator@5 and I changed the reports input: Before: $(Build.SourcesDirectory)/TestResults/*/Coverage/coverage.opencover.xml After: $(Agent.TempDirectory)/*/coverage.opencover.xml