ConsoleRunner, nunit2 format xml blank test results

I’m upgrading our all automate tests to latest nunit, we have ~1100 selenium tests, during full build some of test results xmls’re empty and build fails.

$arguments = `"tests.dll`" --test=`"$testCase`" --work=$work --result=nunit-result-$i.xml;format=nunit2 --framework=net-4.5 --noheader --timeout=1500000 --domain=Multiple"

$process = $(Start-Process $nunit $arguments -PassThru)

When I use new (nunit3) format all xml’s are created as they should.

[ChrisMaddock]

Great - thanks! Let’s close this issue for now, if anyone else is experiencing the same under he current master build, we can reopen.

[aolszowka]

Hi,

We’ve seen this too, we have several (~20) CI Servers running and we see it most often when the Virtual Machine Host (which hosts most of these CI Servers) is under extreme load (IE something kicked off the build).

We thought it was hardware, but this is the first attempt I’ve made to actually dig into NUnit source to see what’s going on and if there is maybe some type of race condition.

I feel I can add some additional information:

• While the file is “empty” its actually been created by the CheckWritability(string) method, I’ve tested this by building a hacked up nunit-v2-result-writer.dll that has tons of additional logging.
• Therefore it my mind it means that somewhere in between ConsoleRunner.cs 188 and 238 we’re dying.

Things I’ve already tried:

• At first I thought there might be some error in nunit-v2-result-writer with an un-handled exception being tossed so I added a bunch of try/catch with additional logging but came up with nothing.
• I have added additional hacks to log when we enter into an exit out of the WriteResultFile(XmlNode,string) method as well as logic around when we enter into WriteXmlOutput(XmlNode,XmlWriter) HOWEVER I did not add any logic around WriteResultFile(XmlNode ,TextWriter)
• I constantly scour our event logs but nothing ever gets written to them. This runs under CI so anything written to the console is long lost (is there anyway to log that out?)

What I can prove:

• When the error occurs we definitely hit CheckWritability(string) (I hacked it up to write out “this is just testing writability” instead of writing an empty file) but at least from what I have seen we never hit WriteXmlOutput(XmlNode,XmlWriter) when we fail.
• If we do get an exception written to the console its not duplicated to the Event Log. It looks like our process might eat up any error coming out of nunit-console

Here’s where I am going next:

• I am going to take a look at our wrapper around nunit-console and make sure we’re respecting any non-zero exit code (we might be getting a useful exception but we’ve lost it???)

Here’s how we’re running the tests: C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe @C:\BuildOutput\Temp\tmp1042.tmp

and here is the content of the tmp1042.tmp:

C:\BuildOutput\TIMSNET\Release\ComputersUnlimited.DatabaseProgrammability.BusinessIntelligence.SSAS.Tests.dll
--x86
--agents=1
--result=C:\BuildOutput\TIMSNET\Release\ComputersUnlimited.DatabaseProgrammability.BusinessIntelligence.SSAS.Tests.dll.TestResult.xml;format=nunit2
--where=cat==Smoke

More background: These tests come from a time when we did not have parallel abilities, therefore they all need to be run sequentially, however as we’ve shifted some of our framework towards running in parallel we changed the runner to be NUnit 3 and just change the arguments (so for tests that we can run in parallel our “response file” is much larger with hundreds of test assemblies). But for the legacy stuff we simply call our logic to create a response file with a single test and a single agent and kick off nunit-console.

However this means that we have 100+ nunit-console instances starting up one right after another in very short order. (I’m sorry I know this is terrible, but its what we’re dealing with).

We only see this when the Virtual Host is under heavy load (all CI’s kick off) so there must be some race condition somewhere that is only getting exposed when the system becomes less responsive.

The tests that seem to fail (its always a different test assembly) are usually extremely small and most times because of the category actually have zero tests to execute, this is probably playing into this (the timing is just way tighter).

I am going to continue digging, but any pointers to where I might go next would be helpful.