Cannot reproduce test results

I’m using FsCheck 2.9 and FsCheck.Xunit 2.9 to test a monadic wrapper around Result type. The complete code is below for inspection. In the last year or so, I’ve seen the test below fail twice (out of thousands of runs on team’s dev machines and build servers) and only once we were able to save the error message containing the seed for value generation. If I use the seed to replay the failing test, the test replay always passes.

This is the full code as used in our codebase:

(* the custom Result type, just a regular DU, not a struct *)
type Result<'result, 'message> =
    | Ok of 'result
    | Error of 'message

(* bind implementation *)
let inline bind (f: 'a -> Result<'b, 'c>) (result: Result<'a, 'c>) =
    match result with
    | Ok x -> f x
    | Error e -> Error e

(* bind infix *)
let inline (>>=) r f = bind f r

(* test aliases since FsCheck doesn't support open generic types *)
type OkType = int
type ErrorType = string

(* the test itself *)
[<Property>]
let associativity
    (f: OkType -> Result<OkType, ErrorType>)
    (g: OkType -> Result<OkType, ErrorType>)
    (result: Result<OkType, ErrorType>) =

    (result >>= f) >>= g = (result >>= (fun v -> (f v) >>= g))

As I said, this test works literally 99.9% of times. The one case where I was able to obtain the full error message is this:

Result Message:
FsCheck.Xunit.PropertyFailedException : Falsifiable, after 47 tests (0 shrinks) (StdGen (242196637,296351952)): Original: (fun:Invoke@3253, fun:Invoke@3253, Ok -12)

But when I tried to run the test manually with the specified StdGen, I wasn’t able to reproduce the failed result. This is what I tried

Check.One({ Config.QuickThrowOnFailure with Replay = Some <| Random.StdGen (242196637,296351952) }, associativity)

I also tried moving the MaxTest and EndSize values from the default of 100 to 10000 and running it few dozen times, but still nothing.

Weirdly enough, there is another property that failed once, also using bind, but it’s part of the result builder tests, so it would be a bit more code to provide here…

Am I doing something wrong when replaying the check? What could possibly be failing there? Is there something wrong with how I’m defining the property in the first place? Or is it a bug in FsCheck’s value generation somehow?

[kurtschelfthout]

Oh, right, thanks. I think FsCheck dep should basically be pinned to = instead of >=. Not sure where the incompatibility comes from though in 2.10 - I updated NUnit, but didn’t touch anything in FsCheck.Xunit.

Anyway don’t see any particular advantage to keeping the >= either, so will change to = .

[kurtschelfthout]

I’ve released 2.10.1 which contains a thread-safe function generator, cherry-picked from fscheck3 branch.

Feel free to re-open if that doesn’t help.