is warning about shrinking chain based arbitrary still valid?

In Transform arbitraries

⚠️ Be aware that the shrinker of such construct might not be able to shrink as much as possible

Is it still valid? Shouldn’t this PR https://github.com/dubzzz/fast-check/pull/113 have made shrinking of chain smarter?

If it’s still valid. Do you have some example case in which chain is resulting in not that good shrink? Just want to know how bad it can be so I can properly evaluate my use of chain

[safareli]

Thanks for the explanation! I have created https://github.com/dubzzz/fast-check/pull/652 to update the docs (feel free to reject it if you have better plans for updating documentation)

[dubzzz]

This warning is unfortunately still true today. The PR #113 enhances the situation but does not solve the whole issue.

Let’s consider the following case: we want to generate a pair of values - (a, b) - such that a is always smaller or equal to b. In order to build that pair we have many alternatives, some based on chain and others on map. Let’s investigate two of them on a predicate that will fail if a and b are too far from each others.

fc.assert(fc.property(
  fc.nat().chain(b => fc.record({ a: fc.nat(b), b: fc.constant(b) })),
  ({a, b}) => b - a <= 5 // the predicate will fail if a and b are not close enough
), {verbose: 2})

/*
My output was:
× [{"a":7,"b":1346021779}]
. √ [{"a":0,"b":0}]
. × [{"a":128450972,"b":673010890}]
. . × [{"a":128450967,"b":336505445}]
. . . × [{"a":128450947,"b":168252723}]
. . . . × [{"a":44324544,"b":84126362}]
. . . . . × [{"a":2261321,"b":42063181}]
. . . . . . × [{"a":2261155,"b":21031591}]
. . . . . . . × [{"a":2260823,"b":10515796}]
. . . . . . . . × [{"a":2260491,"b":5257898}]
. . . . . . . . . × [{"a":2259827,"b":2628949}]
. . . . . . . . . . × [{"a":942695,"b":1314475}]
. . . . . . . . . . . × [{"a":280142,"b":657238}]
. . . . . . . . . . . . × [{"a":274827,"b":328619}]
. . . . . . . . . . . . . × [{"a":89256,"b":164310}]
. . . . . . . . . . . . . . × [{"a":67995,"b":82155}]
. . . . . . . . . . . . . . . × [{"a":24030,"b":41078}]
. . . . . . . . . . . . . . . . × [{"a":607,"b":20539}]
. . . . . . . . . . . . . . . . . × [{"a":9655,"b":10270}]
. . . . . . . . . . . . . . . . . . × [{"a":3363,"b":5135}]
. . . . . . . . . . . . . . . . . . . × [{"a":1966,"b":2568}]
. . . . . . . . . . . . . . . . . . . . × [{"a":342,"b":1284}]
. . . . . . . . . . . . . . . . . . . . . × [{"a":265,"b":642}]
. . . . . . . . . . . . . . . . . . . . . . × [{"a":237,"b":321}]
. . . . . . . . . . . . . . . . . . . . . . . × [{"a":51,"b":161}]
. . . . . . . . . . . . . . . . . . . . . . . . × [{"a":59,"b":81}]
. . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":27,"b":41}]
. . . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":5,"b":21}]
. . . . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":3,"b":11}]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [{"a":6,"b":6}]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [{"a":7,"b":9}]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [{"a":5,"b":10}]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":0,"b":11}]
*/

fc.assert(fc.property(
  fc.tuple(fc.nat(), fc.nat()).map(([v1, v2]) => ({ a: v1 < v2 ? v1 : v2, b: v1 < v2 ? v2 : v1 })),
  ({a, b}) => b - a <= 5 // the predicate will fail if a and b are not close enough
), {verbose: 2})

/*
My output was:
× [{"a":18,"b":493456785}]
. × [{"a":0,"b":18}]
. . √ [{"a":0,"b":0}]
. . × [{"a":0,"b":9}]
. . . √ [{"a":0,"b":5}]
. . . × [{"a":0,"b":7}]
. . . . √ [{"a":0,"b":4}]
. . . . × [{"a":0,"b":6}]
. . . . . √ [{"a":0,"b":3}]
. . . . . √ [{"a":0,"b":5}]
*/

First to notice: map-based implementation goes straight to the smaller failure on shrink. On the other hand chain-based shrinker is more chaotic.

Second thing: chain-based does not report the smaller possible error. Indeed the gap between a and b is larger than expected. With map-based you should always get the minimal failure on this precise example.

Aditionally if you run the chain-based proposal locally you’ll see that shrinker behaves strangely (from an external user point of view) in some situations. Here is an extract of such strange behaviour:

. . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":7,"b":58}]
. . . . . . . . . . . . . . . . . . . . . . . . . . √ [{"a":24,"b":29}]
. . . . . . . . . . . . . . . . . . . . . . . . . . √ [{"a":39,"b":44}]
. . . . . . . . . . . . . . . . . . . . . . . . . . × [{"a":2,"b":51}]

What exactly happens there is fully expected given we used chain. Basically our property failed for {a: 7, b: 58}. Arbitrary produced by chain will behave as follow:

• Let’s try to shrink b (because it drives the chain)
• Next shrink for b is 29. Basically 58 / 2
• Now that we have a new b let’s generate a new a
• Problem: the newly generated a has no special link with the original a. It has been generated from scratch.
• Unfortunately this time a is 24 and fulfills the property

There are several consequences:

• may not shrink towards smaller cases
• may take longer as shrinker depends on your luck (by the way it is still reproducible as the shrinker is using a seeded random generator to build the next a)

The scenario above is basically where chain fails and why we should try to use map or filter instead.

By the way, just to complete the picture, in the scenario described above if we wanted to shrink [{"a":7,"b":58}] while b was already the minimal value, then the shrinker of chain would have shrunk a at constant b. It would have generated [{"a":3,"b":58}].