Tests marked may_fail etc. are shown as failed (red cross)

I am running test cases with doctest that are marked may_fail, should_fail or expected_failures. The asserts within those tests fail, but the test is considered passed due to the decorator. (There is an equivalent feature in Catch called “tags”) In the Test UI, such tests are shown/considered as failed (red cross), when they should be considered as successful (green check). Ideally, this would show some third status indicating that such a decorator was in play (yellow check?), but I don’t know if this is supported upstream.

See for example these doctest tests:

TEST_CASE("expected failures" * doctest::expected_failures(1))
{
   REQUIRE(false);
}
TEST_CASE("may fail" * doctest::may_fail())
{
   REQUIRE(false);
}
TEST_CASE("should fail" * doctest::should_fail())
{
   REQUIRE(false);
}

The output of running such tests in the command line shows the following summary:

[doctest] test cases: 3 | 3 passed | 0 failed | 26 skipped
[doctest] assertions: 3 | 0 passed | 3 failed |
[doctest] Status: SUCCESS!

but in the Test UI I only get red checkmarks:

• Extension Version:
• VS Code Version: 1.60.0
• Catch2 / Google Test / DOCTest Version: doctest 2.4.6
• OS Type and Version: Windows 10
• Using remote-ssh/docker/wsl?: no
• I don’t think that this situtation has been considered so far, so I didn’t think a log would be that useful.

[matepek]

In case of Catch2 they have. Technically it is possible but I guess it’s a bigger investment, would be.

[onqtam]

@matepek

Also it would be great if a doctest issue was opened about reporting the result of SubCases. Don’t know the result of them.

Hmmm, subcases are just {} blocks after if() statements generated from the macros - no return code, and there’s no counting & reporting of failed assertions in subcases, so they don’t have a result.