Issues with custom arbitraries when running tests on NCrunch and TFS.

Hello, I am having really inconsistent behaviour when running on the following test runners:

• TFS
• NCrunch

However, the Resharper test runner is fine.

What the 2 runners that are failing have in common is that they are running tests in parallel in separate processes and the R# runner is running the tests in a single process, AFAIK.

The exception:

Exception during test
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at FsCheck.Runner.checkMethod(Config config, MethodInfo m, FSharpOption`1 target)
   at Xunit.Sdk.ExecutionTimer.Aggregate(Action action)
   at <StartupCode$FsCheck-Xunit>.$PropertyAttribute.testExec@187(PropertyTestCase this, IMessageBus messageBus, Object[] constructorArguments, CancellationTokenSource cancellationTokenSource, XunitTest test, RunSummary summary, TestOutputHelper outputHelper, Unit unitVar0)
System.Exception: The type UnitsNet.Length is not handled automatically by FsCheck. Consider using another type or writing and registering a generator for it.
   at [e-mail](String message)
   at [e-mail](Type t)
   at FsCheck.Common.f@1[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n, Unit _arg1)
   at FsCheck.Common.memoizeWith[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n)
   at [e-mail]_Generator()
   at FsCheck.Arbitrary`1.FsCheck-IArbitrary-get_GeneratorObj()
   at [e-mail](IEnumerable`1& next)
   at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at Microsoft.FSharp.Collections.SeqModule.ToArray[T](IEnumerable`1 source)
   at FsCheck.ReflectArbitrary.productGen@63[a](FSharpFunc`2 getGenerator, IEnumerable`1 ts, FSharpFunc`2 create)
   at FsCheck.ReflectArbitrary.reflectObj(FSharpFunc`2 getGenerator, Type t)
   at [e-mail](Type t)
   at FsCheck.Common.f@1[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n, Unit _arg1)
   at FsCheck.Common.memoizeWith[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n)
   at [e-mail]_Generator()
   at FsCheck.Testable.forAll[a,b](Arbitrary`1 arb, FSharpFunc`2 body)
   at FsCheck.Runner.check[a](Config config, a p)

or

System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
---- System.Exception : The type AwtSG.Domain.NavigationUtilities.Latitude is not handled automatically by FsCheck. Consider using another type or writing and registering a generator for it.
Stack trace
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at FsCheck.Runner.checkMethod(Config config, MethodInfo m, FSharpOption`1 target)
at <StartupCode$FsCheck-Xunit>.$PropertyAttribute.testExec@187(PropertyTestCase this, IMessageBus messageBus, Object[] constructorArguments, CancellationTokenSource cancellationTokenSource, XunitTest test, RunSummary summary, TestOutputHelper outputHelper, Unit unitVar0)
----- Inner Stack Trace -----
at [e-mail](String message)
at [e-mail](Type t)
at FsCheck.Common.f@1[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n, Unit _arg1)
at FsCheck.Common.memoizeWith[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n)
at [e-mail]_Generator()
at FsCheck.Arbitrary`1.FsCheck-IArbitrary-get_GeneratorObj()
at [e-mail](IEnumerable`1& next)
at Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1.MoveNextImpl()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at Microsoft.FSharp.Collections.SeqModule.ToArray[T](IEnumerable`1 source)
at FsCheck.ReflectArbitrary.productGen@63[a](FSharpFunc`2 getGenerator, IEnumerable`1 ts, FSharpFunc`2 create)
at FsCheck.ReflectArbitrary.reflectObj(FSharpFunc`2 getGenerator, Type t)
at [e-mail](Type t)
at FsCheck.Common.f@1[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n, Unit _arg1)
at FsCheck.Common.memoizeWith[a,b](IDictionary`2 memo, FSharpFunc`2 f, a n)
at [e-mail]_Generator()
at FsCheck.Testable.forAll[a,b](Arbitrary`1 arb, FSharpFunc`2 body)
at FsCheck.Runner.check[a](Config config, a p)

(The second one is taken from a TFS build of a different project)

A minimum working example, which I can reproduce the error with consistently is below.

The .cs file:

using FsCheck;
using FsCheck.Xunit;
using UnitsNet;
using Xunit;

namespace FsCheckNcrunchExample
{
    public class UnitTest1
    {
        public class Arbitraries
        {
            public static Arbitrary<Length> Lengths()
                => Arb.Default.Float()
                    .Filter(i => i > 0 && i < 100)
                    .Convert(Length.FromMeters, i => i.Meters);
        }

        static UnitTest1()
        {
            Arb.Register<Arbitraries>();
        }

        [Property]
        public void Test1(Length a, Length b)
        {
            Assert.True(true);
        }

        [Property]
        public void Test2(Length a, Length b)
        {
            Assert.True(true);
        }

        [Property]
        public void Test3(Length a, Length b)
        {
            Assert.True(true);
        }

        [Property]
        public void Test4(Length a, Length b)
        {
            Assert.True(true);
        }
    }
}

The project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0-preview-20170425-07" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
    <PackageReference Include="FsCheck.Xunit" Version="*" />
    <PackageReference Include="UnitsNet" Version="*" />
  </ItemGroup>

</Project>

The dependency tree in Visual Studio:

[kurtschelfthout]

Going to close this as the issue is resolved, please don’t let that stop you from contributing an example or even better documentation!

[aignas]

Actually, if I register the arbitraries in the non static constructor, then it is working more reliably. Actually, on NCrunch for my example project it has not failed.