skipAllAfterTimeLimit throws an error instead of skipping tests

🐛 Bug Report

skipAllAfterTimeLimit throws an error instead of skipping the remaining tests like the name suggests

To Reproduce

Steps to reproduce:

    it(‘should skip tests’, async () => {
        jest.useRealTimers();
        const property = fc.asyncProperty(
            fc.constant(''),
            async () => {
                await new Promise(resolve => setTimeout(resolve, 100));
                expect(true).toBe(true);
            }
        );
        await fc.assert(property, {
            numRuns: 10,
            skipAllAfterTimeLimit: 1,

        });
    });
});

Expected behavior

It should skip all remaining tests instead of throwing error. We could have a parameter, similar to maxSkipsPerRun, to specify the max amount of tests it can skip.

Your environment

Packages / Softwares	Version(s)
fast-check	1.16.2
node	10.14.1
TypeScript	3.4.5

[ZeRego]

I see that the fail message with very verbose mode only shows the generated value. Could we add the seed and path to each one ?

Current error message:

Error: Failed to run property, too many pre-condition failures encountered

Ran 9 time(s)
Skipped 1001 time(s)

Execution summary:
√ [""]
√ [""]
√ [""]
√ [""]
√ [""]
√ [""]
√ [""]
√ [""]
√ [""]
! [""]
! [""]
...
! [""]

Hint (1): Try to reduce the number of rejected values by combining map, flatMap and built-in arbitraries
Hint (2): Increase failure tolerance by setting maxSkipsPerRun to an higher value

[dubzzz]

skipAllAfterTimeLimit

I agree that skipAllAfterTimeLimit was not well defined. The initial idea behind this feature was: mark runs as skipped as soon as the time limit expired.

In fast-check, a run can have 3 different states: success, failure, skipped (mainly triggered by fc.pre). Initially skipAllAfterTimeLimit was supposed to mark all runs as skipped after the time limit.

There are two possible outcomes:

• timeout occurs during shrinking: shrinking will be interrupted by marking all the shrunk values as skipped
• timeout occurs while there are remaining values to generate: test will fail because it will never be able to reach the required number of tests

See https://github.com/dubzzz/fast-check/pull/425 which just fix the feature as it was initially defined.

interruptAfterTimeLimit

On the contrary, interruptAfterTimeLimit does not mark runs as skipped. It just stops the execution of the test suite. There two possibilities: either it should be marked as a failure, or a success with not enough runs (see markInterruptAsFailure in https://github.com/dubzzz/fast-check/pull/428).

Internally interruptAfterTimeLimit really interrupts the execution either by marking it as failure or as success. While skipAllAfterTimeLimit just marks everything as skipped.

conclusion

• skipAllAfterTimeLimit: mark runs as skipped after timeout (shrunk values will still be produced)
• interruptAfterTimeLimit: stop execution after timeout - test is considered as success if it has never encountered an issue - I have to double check in my implementation that if it happens while shrinking the reported error is the smallest failure not the shrunk where it stopped
• interruptAfterTimeLimit and markInterruptAsFailure:true: stop execution after timeout - an interrupted test is considered as a failed test - maybe the contrary markInterruptAsSuccess

Actually I’m still looking for the right way to add this interrupt/skip/whatever feature (maybe replacing skipAllAfterTimeLimit).