Threshold value not working

Hi,

I just upgraded to the latest v1.1.0 GA build. I’m using the new /p:Threshold=80 option to the build process yet the test run still returns ‘successful’ even if the coverage is below the threshold.

This is the command I run to build:

dotnet test --configuration Release --logger ‘trx;logfilename=mstest-results.trx’ /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutputDirectory=TestResults /p:Threshold=80

This is the output:

Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:01.5821864]   Discovering: UnitTests
[xUnit.net 00:00:01.7179983]   Discovered:  UnitTests
[xUnit.net 00:00:01.7245869]   Starting:    UnitTests
[xUnit.net 00:00:01.9276830]     AuthResponsesOperationFilter does not add 401 response to operation without AuthoriseAttribute [SKIP]
[xUnit.net 00:00:01.9282262]       Cannot yet mock ApiDescription objects
Skipped  AuthResponsesOperationFilter does not add 401 response to operation without AuthoriseAttribute
Standard Output Messages:
 Cannot yet mock ApiDescription objects
[xUnit.net 00:00:01.9862873]     AuthResponsesOperationFilter Adds 401 response to operation with AuthoriseAttribute [SKIP]
[xUnit.net 00:00:01.9863816]       Cannot yet mock ApiDescription objects
Skipped  AuthResponsesOperationFilter Adds 401 response to operation with AuthoriseAttribute
Standard Output Messages:
 Cannot yet mock ApiDescription objects
[xUnit.net 00:00:02.3872529]   Finished:    UnitTests

Total tests: 33. Passed: 31. Failed: 0. Skipped: 2.
Test Run Successful.
Test execution time: 3.8409 Seconds
WARNING: Overwriting results file: /home/jenkins/agent/workspace/an-updated-coverage-process-7N7FVRFUDPWSNYEY7B7OE6IM5VJ26KCY5NFVLTNHW4HURJ65KZAQ/tests/UnitTests/TestResults/mstest-results.trx
Results File: /home/jenkins/agent/workspace/an-updated-coverage-process-7N7FVRFUDPWSNYEY7B7OE6IM5VJ26KCY5NFVLTNHW4HURJ65KZAQ/tests/UnitTests/TestResults/mstest-results.trx

Calculating coverage result...
  Generating report 'TestResults/coverage.xml'

| Module                 | Coverage |
|------------------------|----------|
| ApplicationCore        | 75%      |
| Infrastructure         | 37.8%    |
| Nedbank.Authentication | 0%       |
| Web                    | 42.9%    |

Am I missing something or is this not working right now.

Also my build environment is:

Microsoft ® Build Engine version 15.3.409.57025 for .NET Core We’re running .NET Core 2.0 on Linux. (Install .NET Core SDK 2.1.104 Package)

[daveMueller]

@DhavalAlbiorix the issue with msbuild was fixed by the microsoft guys with sdk 7.0.101 that was released last week https://github.com/microsoft/vstest/issues/4014. I just checked it with a repro from another related issue and it works fine. Please give it a try again.

[petli]

@DhavalAlbiorix Sorry, I forgot about the threshold part, I was just thinking about the command line parameter passing. The vstest driver doesn’t support thresholds, you would need to extend the CI script to process the reports from coverlet with e.g. https://reportgenerator.io/ and check that in the script.