`AutoFaker<T>.Generate(null)` Fails on Ubuntu

I have a project that I’m building and I’m using Bogus + AutoBogus in my test suites. I personally develop on macOS and Windows and haven’t experienced the issue on either, but when I use Ubuntu on Azure Pipelines, calling AutoFaker<T>.Generate(null) (or Generate() for that matter, they amount to the same), Generate() throws:

Failed   Heroku.NET.Tests.Apps.AppsClientTests.DisableACMSendsTheCorrectRequest
Error Message:
 System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at Bogus.Faker`1.FinishWith(Action`2 action) in C:\projects\bogus\Source\Bogus\Faker[T].cs:line 376
   at AutoBogus.AutoFaker`1.PrepareFinish(AutoGenerateContext context)
   at AutoBogus.AutoFaker`1.Generate(String ruleSets)
   at Heroku.NET.Tests.Apps.MockApp.Create(String ruleset) in /app/test/Heroku.NET.Tests/Apps/MockApp.cs:line 26
   at Heroku.NET.Tests.Apps.AppsClientTests.DisableACMSendsTheCorrectRequest() in /app/test/Heroku.NET.Tests/Apps/AppsClientTests.cs:line 97
--- End of stack trace from previous location where exception was thrown ---

If, instead, I use Generate("default") to accomplish the same effet, everything works as expected and as they do on macOS and Windows. I’m not sure what the difference could be, unless MSCorlib has some weird implementation detail on Linux that makes the null string a problem.

You can see this in action on my current working branch where b3a5946 exhibits the issue and aa19cd0 fixes it.

I’m using .NET Core 2.2.102 on macOS 10.14.1, Windows 10 1809, and Ubuntu 16.04.

[bchavez]

Hey there,

I was able to reproduce the issue on Ubuntu 16.04 even at the latest commit:

aa19cd0d0ae77045f194c0d1bafba959bc23f277
Author: berwyn <[e-mail]>
Date:   Tue Jan 22 20:48:07 2019 -0500

    Fix NullPointer on Linux

Heroku.NET.Tests.Connections.HerokuV3ConnectionTests.TestPostRequestGeneration [FAIL]
Hosting environment: Production
Content root path: /home/brian/test/heroku-dotnet/test/Heroku.NET.Tests/bin/Debug/netcoreapp2.1/
Now listening on: http://0.0.0.0:9095
Failed   Heroku.NET.Tests.Connections.HerokuV3ConnectionTests.TestPostRequestGeneration
Error Message:
 System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at Bogus.Faker`1.FinishWith(Action`2 action) in C:\projects\bogus\Source\Bogus\Faker[T].cs:line 376
   at AutoBogus.AutoFaker`1.PrepareFinish(AutoGenerateContext context)
   at AutoBogus.AutoFaker`1.Generate(String ruleSets)
   at Heroku.NET.Tests.Apps.MockApp.Create(String ruleset) in /home/brian/test/heroku-dotnet/test/Heroku.NET.Tests/Apps/MockApp.cs:line 27
   at Heroku.NET.Tests.Connections.HerokuV3ConnectionTests.TestPostRequestGeneration() in /home/brian/test/heroku-dotnet/test/Heroku.NET.Tests/Connections/HerokuV3ConnectionTests.cs:line 77
--- End of stack trace from previous location where exception was thrown ---

Total tests: 11. Passed: 10. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 3.2518 Seconds

As @nickdodd79 mentioned, the issue is likely the parallelism with xunit.

Multi-threaded concurrent access to a global static Faker<T> in MockApp.cs is likely the cause of the issue. As Nick also mentioned, the dictionaries used under the hood are basic Dictionary<T> and aren’t thread-safe. Threading issues are also usually a sign of the flaky behavior that you experienced with tests running okay on OSX and Windows, but not Linux probably due to differences in thread-scheduling.

So, there are a couple of solutions for us.

1. Use a Lazy<T> for thread-safe initialization of the global static Faker<T> object.
2. Create a dedicated instance of Faker<T> for each test-request of a Faker<T>

I’d probably recommend the 2nd option because you might want to mutate the Faker<T> more specifically later down the road for some specialized test and you don’t want those mutations leaking into other tests as they would with a mutated global static object.

Details are below:

1. Use Lazy<T> to create the global static.

internal static class MockApp
{
    private static Lazy<Faker<App>> lazyFaker = new Lazy<Faker<App>>(CreateAutoFaker, isThreadSafe:true);

    private static Faker<App> CreateAutoFaker()
    {
        var _faker = new AutoFaker<App>()
            .RuleFor(a => a.GitUrl, f => f.Internet.Url())
            .RuleFor(a => a.CreatedAt, f => f.Date.Past())
            .RuleFor(a => a.UpdatedAt, f => f.Date.Recent())
            .RuleFor(a => a.ArchivedAt, () => null);

        _faker.RuleSet("archived", rules =>
        {
            rules.RuleFor(a => a.ArchivedAt, f => f.Date.Recent());
        });
        return _faker;
    }

    internal static App Create(string ruleset = "default")
    {
        return lazyFaker.Value.Generate(ruleset);
    }
}

1. Dedicated object for each request for a Faker<T>. Might be better to just return Faker<App> instead of App so you can mutate Faker<T> a little more for specialized unit tests.

internal static class MockApp
{
    internal static App Create(string ruleset = "default")
    {
        var _faker = new AutoFaker<App>()
            .RuleFor(a => a.GitUrl, f => f.Internet.Url())
            .RuleFor(a => a.CreatedAt, f => f.Date.Past())
            .RuleFor(a => a.UpdatedAt, f => f.Date.Recent())
            .RuleFor(a => a.ArchivedAt, () => null);

        _faker.RuleSet("archived", rules =>
        {
            rules.RuleFor(a => a.ArchivedAt, f => f.Date.Recent());
        });
        return _faker.Generate(ruleset);
    }
}

Also, check out the .Clone() method on Faker<T> for some more interesting test scenarios. 😎

And when you make the fix you should have some good success 🎉

...
Total tests: 11. Passed: 11. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 3.1639 Seconds
brian@linux:~/test/heroku-dotnet/test/Heroku.NET.Tests$ 

Hope that helps!

Thanks,
Brian

🚗 🚙 “Let the good times roll…”

[berwyn]

Thanks for that excellent breakdown @bchavez, this is my first foray into xunit and I hadn’t even thought about the concurrent access! Clearly this is a problem with my use of AutoBogus and not the library itself, so I’ll go ahead and close the issue.

Thanks both for taking the time to help out!