FsCheck turns 3.0?

This issue is to discuss ideas or changes for FsCheck 3.0. I started thinking about this starting from existing issues and recurring questions that indicate more problematic parts; so far there aren’t very really new features in here - mostly a rewrite or update of existing ones.

Where I have worked on these things I’ve included the branch name; these branches won’t necessarily work or even compile. Sometimes they are just broad strokes to play with the idea.

EDIT: all development marked done below is merged to the fscheck3 branch.

• branch replace-random Replace the random number generation by the algorithm in “Fast splittable pseudo-random number generators” by Steel et al. which provides a better basis to do Splittable PRNG. The current implementation taken from Hugs has been shown to have serious statistical defects. Also FsCheck doesn’t need to rely on splitting quite so much, if a generator also returns the seed after use.
  • Figure out why it’s slower (tuple allocations, fixed)
• branch Gen-CPS even though function using Gen may be tail-recursive, that doesn’t necessarily mean the overall generator is tail recursive. Using a CPS style representation of Gen can solve some of these issues. This change is relatively straightforward I think, and can be made without changing any part of the API except the Gen constructor.
  • This is A LOT slower. Figure out why, may have to can this one…
• branch no-static-arb Remove the Arbitrary registration mechanism. (Perhaps keep an immutable one just keeping the default generators that ship with FsCheck). Replace with explicitly passing Arbitrary instance (e.g. FsList takes an Arbitrary<'a> to generate the elements with). We’ll still need some kind of map 'a -> Arbitrary<'a> to parametrize the reflective Arbitrary instance with, so it’s likely the “typeclass” discovery mechanism is still useful. I feel like this is perhaps slightly more verbose, but will remove a bunch of confusion and implicit “magic”. In addition, it’s hard to maintain some kind of consistency when tests can for example run concurrently, because it relies on modifying global static values.
• Re-think how we handle running tests. I’ve been wondering if we want to keep maintaining integration with test running frameworks. From a maintenance perspective (and given all the issues on how to interopate with xunit/nunit and all their versions, perhaps from a user perspective too) it would certainly be a lot simpler if FsCheck would just be a library. For easy integration, we could reverse the defaults: Check.Quick could throw an exception by default, we can remove Check.QuickThrowOnFailure and add Check.QuickNoThrow. I’d also like some to expose running tests more explicitly, perhaps as an IEnumerable of test/shink results instead of the callback based interface it is now; this would hopefully allow things like storing the results in a DB or whatever we may want to add later on.
• Remove some of the assertion-like property combinators, like Prop.within. Users tend to use the assertions from some more focused framework, no real point in maintaining these here really.
• Maybe remove the possibility of returning bool to signify success/failure, standardize on throwing exceptions. This is standard practice anyway for .NET based testing tools; it removes a bunch of overloads on the C# side that the C# compiler is not super happy with; it may be possible to get rid of label (as the test failure is propagated via the exception message).

Any thoughts on these or further ideas are appreciated.

[ploeh]

FWIW, I’ve suggested an alternative for handling custom Arbitraries in #334. This would, AFAICT, provide a simple and decoupled, Reflection-based mechanism for custom types.

Remaining is figuring out how to deal with pre-defined types like string, bool, int64 and so on.

How about removing the Arbitrary label from FsCheck.Config and replacing it with a dictionary of well-known types? This would also provide an explicit extensibility mechanism for users.

[kurtschelfthout]

@theimowski yes, in v3 all the obsoleted stuff will go. But I intend to do that at the end to avoid too many merge problems. I am merging from master to v3 branch quite regularly. In case you are wondering, no need to send a PR for that.