Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

"Execution terminated after first error" does not fail the console runner

See original GitHub issue

After updating to NUnit 3.7.1 (Console runner 3.7.0) I noticed that some of my tests are no longer executed. I use the option to stop on first failure. When analyzing the log file I found out that there was an exception in tear down which was reported correctly in the console but the console runner itself does not report failure. Is it intended that tear down (and maybe also setup) exceptions no longer fail the test run?

Details:

5) TearDown Error : SomeIntegrationTest.Test<Enum1>(Param1,False,True)
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
--TearDown
   at SomeIntegrationTest....

Execution terminated after first error

Run Settings
    RuntimeFramework: net-4.5
    DisposeRunners: True
    WorkDirectory: C:/Teamcity/BuildAgent/work/52362ff0ee379106/bin/x64/Debug
    StopOnError: True
    MaxAgents: 1
    NumberOfTestWorkers: 1
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False

Test Run Summary
  Overall result: Failed
  Test Count: 1840, Passed: 621, Failed: 0, Warnings: 0, Inconclusive: 200, Skipped: 1019
    Skipped Tests - Ignored: 5, Explicit: 1014, Other: 0
  Start time: 2017-08-18 12:59:21Z
    End time: 2017-08-18 13:09:08Z
    Duration: 586.926 seconds

Issue Analytics

State:
Created 6 years ago
Comments:8 (6 by maintainers)

Top GitHub Comments

1reaction

Sam13commented, Aug 21, 2017

I had to strip some stuff out, but should be sufficient to fix the issue IntegrationTest.x64.Release.TestResult.txt

0reactions

CharliePoolecommented, Aug 21, 2017

TestResult seems to confirm my guess.

We should decide what the return code should be in these cases. One approach would be to return the number of errors and failures displayed, even if they are not test case errors. Another would be to create a new negative return code value indicating that onetime setup or teardown errors occured even though all test cases passed. A third simple approach would be to return the existing “unexpected error” code.