Flaky tests when run in parallel

Hello,

We use FakeItEasy, xUnit and Rebus at my company. We have some tests that are flaky and I have looked into why this is. My current best guess is that there is some kind of odd problem with concurrency in FakeItEasy which is why I’m creating this bug report. I have been able to reduce our code down to a small toy example that still is flaky Flaky.zip. I understand that you might not know about Rebus so I will try my best to explain what I think is happening without going into too much unnecessary detail.

We use Rebus.Testhelpers to create a mock version of our “saga”. That fake saga receives a command message that it is supposed to handle. The handler for the message changes the state of the fake Saga. Somewhere along the line, the fake Saga tries to clone its state by serializing the “saga data” to JSON and deserializing it back to saga data again. This is where an exception sometimes (!) is thrown which can be seen by going through the LogEvents of the saga. The exception is thrown at this line https://github.com/rebus-org/Rebus.TestHelpers/blob/7ebb5f1b968b34f69dd1afffd55f6197afc103e0/Rebus.TestHelpers/Internals/InMemorySagaStorage.cs#L238.

The exception is:

Newtonsoft.Json.JsonSerializationException

Could not find type 'Castle.Proxies.SomeOtherDataProxy' in assembly 'DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=a621a9e7e5c32e69'.

Note again that this happens sometimes. When it works, the type is properly set to the real type instead of the proxy type and the deserialization works fine.

This seems like some kind of problem with concurrency. I read up on parallelism in xUnit and sure enough, adding the class attribute [Collection("Flaky tests")] to both of the test classes eliminates the flakiness.

Converting the test project to MSTest also eliminates the flakiness. I suspect this is because MSTest does not execute tests in parallel by default.

Commenting out line 12 in MySecondTestClass.cs also eliminates the flakiness.

At this point I’m stuck. I need someone smarter than me to look at this to get any further. Could this be an issue with xUnit? Any help is appreciated, thank you very much 😃

[blairconrad]

Thanks, @fishie, yup, your summary seems good. The only thing I’d possibly add is that I think that Castle.Core creates the assemblies with a public key when it’s asked to proxy a type from an assembly that itself has a public key (as object is), and otherwise will not. I’ve not tried this out, but my guess is that if you instead strong-named the assembly that contained SomeOtherData, the problem may also go away.

Okay. Since FakeItEasy doesn’t seem to be the problem, I think we can close this issue. Thanks for raising it, fishie. Happy FakeItEasying and God Jul.

[blairconrad]

Hey! Remember my guess about the duplicate assemblies, differing only by public key, causing the problem? It’s been more than “a couple of days”, but I got back into it. I was right. 😮

Anyhow, this change to Rebus.TestHelpers

diff --git a/Rebus.TestHelpers/Internals/InMemorySagaStorage.cs b/Rebus.TestHelpers/Internals/InMemorySagaStorage.cs
index f1d04f7..c234075 100644
--- a/Rebus.TestHelpers/Internals/InMemorySagaStorage.cs
+++ b/Rebus.TestHelpers/Internals/InMemorySagaStorage.cs
@@ -23,7 +23,8 @@ namespace Rebus.TestHelpers.Internals

         readonly JsonSerializerSettings _serializerSettings = new JsonSerializerSettings
         {
-            TypeNameHandling = TypeNameHandling.All
+            TypeNameHandling = TypeNameHandling.All,
+            TypeNameAssemblyFormatHandling = TypeNameAssemblyFormatHandling.Full,
         };

         internal IEnumerable<ISagaData> Instances

makes InMemorySagaStorage.Clone produce

{
    "$type": "MyProject.MySagaData, MyProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
    "SomeData": {
        "$type": "MyProject.SomeData, MyProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
        "Id": null
    },
    "SomeOtherData": {
        "$type": "Castle.Proxies.SomeOtherDataProxy, DynamicProxyGenAssembly2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null"
    },
    "Id": "2f8ba26d-636f-44c1-8f85-b450baebca3d",
    "Revision": 0
}

And then my test passes.

I suppose one could make the case that it’s weird for Castle.Core (and by extension FakeItEasy) to produce two assemblies, differing only by public key, but I’m not sure anything would change. Me, I’d consider pursuing a fix in Rebus.TestHelpers.