Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
UnitTest randomly failing
See original GitHub issueThere is a very strange behavior with our unit tests. One of them seems to randomly fail without any helpful error message, plus the overall result is passed.
This is the result from the console: Test Run Summary Overall result: Passed Test Count: 2551, Passed: 2550, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0 Failed Tests - Failures: 1, Errors: 0, Invalid: 0 Start time: 2018-08-14 09:28:56Z End time: 2018-08-14 09:29:56Z Duration: 60.814 seconds
Results (nunit3) saved as build\UnitTest\TestResults.xml Running build failed. Error: NUnit test failed (1).
But in the TestRusults.xml from the same job everything seems to be ok and all 2551 tests have passed. So something clearly doesn’t add up here.
Here is a snippet from the TestResults.xml:
<?xml version="1.0" encoding="utf-8" standalone="no"?> <test-run id="2" testcasecount="2551" result="Passed" total="2551" passed="2551" failed="0" inconclusive="0" skipped="0" asserts="3830" engine-version="3.8.0.0" clr-version="4.0.30319.42000" start-time="2018-08-14 09:28:56Z" end-time="2018-08-14 09:29:56Z" duration="60.813952">
Is this a known issue? Is there a workaround or anything we can do to prevent the test from randomly failing?
Issue Analytics
- State:
- Created 5 years ago
- Comments:8 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
I think the “double call” of nunit3-console is just an overly eager logging from the FAKE (F# MAKE) NUnit 3 target, but we’ll investigate on that as well.
It’s strange though, that the XML file reports all tests as passed but lists a failed test in the details, thanks for pointing that out. We’ll try to get down to the essence of the problem and hopefully can provide something helpful.
Just to clarify: It’s not a randomly failing test, but the console runner randomly failing. As you can see in the console summary, it states
The XML is:
And this is for the same run. It looks like there might be a race condition inside the NUnit runner that causes the last test not to be tracked correctly.
If there really is a failing test, the test name is written to the console, but when this happens, it does not name any failed tests.
We hope that you have an idea what we could try to either give you more information or work around this issue.