Property Test Discussion

I am working on overhauling our property testing as part of the upcoming 4.0 release. To this end, I have brought together the requirements (based off tickets existing in this tracker), and come up with a basic design. I would like feedback on this design before I fully implement it. There are also some questions that I don’t have an answer to yet that I would like to discuss. At this stage everything is open to change.

Property Test Requirements

Deterministic Re-runs

If a test failed it is useful to be able to re-run the tests with the same values. Especially in cases where shrinking is not available. Therefore, the test functions accept a seed value which is used to create the Random instance used by the tests. This seed can then be programatically set to re-run the tests with the same random instance.

By default the seed is null, which means the seed changes each time the test is run.

Exhaustive

The generators are passed an Exhaustivity enum value which determines the behavior of generated values.

• Random - all values should be randomly generated
• Exhaustive - every value should be generated at least once. If, for the given iteration count, all values cannot be generated, then the test should fail immediately.
• Auto - Exhaustive if possible and supported, otherwise random.

By default Auto mode is used.

Question - do we want to be able to specify exhaustivity per parameter?

In #1101 @EarthCitizen talks about Series vs Gen. I do like the nomenclature, but we would need another abstraction (Arbitrary?) on top which would then be able to provide a Gen or Series as required based on the exhaustive flag.

Question - do we want to implement this way as opposed to the way I have outlined in the code below

Min and Max Successes

These values determine bounds on how many tests should pass. Typically min and max success would be equal to the iteration count, which gives the forAll behavior. For forNone behavior, min and max would both be zero. Other values can be used to mimic behavior like forSome, forExactly(n) and so on.

By default, min and max success are set to the iteration count.

Distribution

It is quite common to want to generate values across a large number space, but have a bias towards certain values. For example, when writing a function to test emails, it might be more useful to generate more strings with a smaller number of characters than larger amounts. Most emails are probably < 50 characters for example.

The distribution mode can be used to bias values by setting the bound from which each test value is generated.

• Uniform - values are distributed evenly across the space. For an integer generator of values from 1 to 1000 with 10 runs, a random value would be generated from 0.100, another from 101…200 and so on.
• Pareto - values are biased towards the lower end on a roughly 80/20 rule.

By default the uniform distribution is used.

The distribution mode may be ignored by a generator if it has no meaning for the types produced by that generator.

The distribution mode has no effect if the generator is acting in exhaustive mode.

Question - it would be nice to be able to specify specific “biases” when using specific generators. For example, a generator of A-Z chars may choose to bias towards vowels. How to specify this when distribution is a sealed type? Use an interface and allow generators to create their own implementations?

Shrinking Mode

The ShrinkingMode determines how failing values are shrunk.

• Off - Shrinking is disabled for this generator
• Unbounded - shrinking will continue until the minimum case is reached as determined by the generator
• Bounded(n) - the number of shrink steps is capped at n. After this, the shrinking process will halt, even if the minimum case has not been reached. This mode is useful to avoid long running tests.

By default shrinking is set to Bounded(1000).

Question1 - do we want to be able to control shrinking per parameter? Turn it off for some parameters, and not others?

When mapping on a generator, shrinking becomes tricky. If you have a mapper from GenT to GenU and a value u fails, you need to turn that u back into a t, so you can feed that t into the original shrinker. So you can either keep the association between the original value and mapped value, or precompute (lazily?) shinks along with the value.

Question2 - which is the best approach?

Gen design

The gens accept the Random instance used for this run. They accept an iterations parameter so they know the sample space when calculating based on a distribution They accept the exhausitivity mode and the distribution mode. Question - move the iteration count into the distribution parameter itself?

Note that the gens no longer specify a shrinker, but should provide the shrinks along with the value (see shrinker section for discussion).

/**
 * A Generator, or [Gen] is responsible for generating data* to be used in property testing.
 * Each generator will generate data for a specific type <T>.
 *
 * The idea behind property testing is the testing framework will automatically test a range
 * of different values, including edge cases and random values.
 *
 * There are two types of values to consider.
 *
 * The first are values that should usually be included on every test run: the edge cases values
 * which are common sources of bugs. For example, a function using [Int]s is more likely to fail
 * for common edge cases like zero, minus 1, positive 1, [Int.MAX_VALUE] and [Int.MIN_VALUE]
 * rather than random values like 159878234.
 *
 * The second set of values are random values, which are used to give us a greater breadth to the
 * test cases. In the case of a functioin using [Int]s, these random values could be from across
 * the entire integer number line.
 */
interface Gen<T> {

   /**
    * Returns the values that are considered common edge case for this type.
    *
    * For example, for [String] this may include the empty string, a string with white space,
    * a string with unicode, and a string with non-printable characters.
    *
    * The result can be empty if for type T there are no common edge cases.
    *
    * @return the common edge cases for type T.
    */
   fun edgecases(): Iterable<T>

   /**
    * Returns a sequence of values to be used for testing. Each value should be provided together
    * with a [Shrinker] to be used if the given value failed to pass.
    *
    * This function is invoked with an [Int] specifying the nth test value.
    *
    * @param random the [Random] instance to be used for random values. This random instance is
    * seeded using the seed provided to the test framework so that tests can be deterministically rerun.
    *
    * @param iterations the number of values that will be required for a successful test run.
    * This parameter is provided so generators know the sample space that will be required and can thus
    * distribute values accordingly.
    *
    * @param exhaustivity specifies the [Exhaustivity] mode for this generator.
    *
    * @param distribution specifies the [Distribution] to use when generating values.
    *
    * @return the test values as a lazy sequence.
    */
   fun generate(
      random: Random,
      iterations: Int,
      exhaustivity: Exhaustivity = Exhaustivity.Auto,
      distribution: Distribution
   ): Sequence<Pair<T, Shrinker<T>>>

   companion object
}

fun Gen.Companion.int(lower: Int, upper: Int) = object : Gen<Int> {
   private val literals = listOf(Int.MIN_VALUE, Int.MAX_VALUE, 0)
   override fun edgecases(): Iterable<Int> = literals
   override fun generate(
      random: Random,
      iterations: Int,
      exhaustivity: Exhaustivity,
      distribution: Distribution
   ): Sequence<Pair<Int, Shrinker<Int>>> {

      val randomized = infiniteSequence { k ->
         val range = distribution.get(k, iterations, lower.toLong()..upper.toLong())
         random.nextLong(range).toInt()
      }

      val exhaustive = generateInfiniteSequence {
         require(iterations <= upper - lower)
         (lower..upper).iterator().asSequence()
      }.flatten()

      val seq = when (exhaustivity) {
         Exhaustivity.Auto -> when {
            iterations <= upper - lower -> exhaustive
            else -> randomized
         }
         Exhaustivity.Random -> randomized
         Exhaustivity.Exhaustive -> exhaustive
      }
      return seq.map { Pair(it, IntShrinker) }
   }
}

fun <T, U> Gen<T>.map(f: (T) -> U): Gen<U> {
   val outer = this
   return object : Gen<U> {
      override fun edgecases(): Iterable<U> = outer.edgecases().map(f)
      override fun generate(
         random: Random,
         iterations: Int,
         exhaustivity: Exhaustivity,
         distribution: Distribution
      ): Sequence<Pair<U, Sequence<U>>> =
         outer.generate(random, iterations, exhaustivity, distribution)
            .map { (value, shrinks) ->
               Pair(f(value), shrinks.map { f(it) }.asSequence())
            }
   }
}

sealed class Distribution {

   abstract fun get(k: Int, iterations: Int, range: LongRange): LongRange

   /**
    * Splits the range into discrete "blocks" to ensure that random values are distributed
    * across the entire range in a uniform manner.
    */
   object Uniform : Distribution() {
      override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
         val step = (range.last - range.first) / iterations
         return (step * k)..(step * (k + 1))
      }
   }

   /**
    * Values are distributed according to the Pareto distribution.
    * See https://en.wikipedia.org/wiki/Pareto_distribution
    * Sometimes referred to as the 80-20 rule
    *
    * tl;dr - more values are produced at the lower bound than the upper bound.
    */
   object Pareto : Distribution() {
      override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
         // this isn't really the pareto distribution so either implement it properly, or rename this implementation
         val step = (range.last - range.first) / iterations
         return 0..(step * k + 1)
      }
   }
}

sealed class Exhaustivity {

   /**
    * Uses [Exhaustive] where possible, otherwise defaults to [Random].
    */
   object Auto : Exhaustivity()

   /**
    * Forces random generation of values.
    */
   object Random : Exhaustivity()

   /**
    * Forces exhausive mode.
    */
   object Exhaustive : Exhaustivity()
}

sealed class ShrinkingMode {

   /**
    * Shrinking disabled
    */
   object Off : ShrinkingMode()

   /**
    * Shrinks until no smaller value can be found. May result in an infinite loop if shrinkers are not coded properly.
    */
   object Unbounded : ShrinkingMode()

   /**
    * Shrink a maximum number of times
    */
   data class Bounded(val bound: Int) : ShrinkingMode()
}

[sksamuel]

Here is another implementation proposal based on feedback received so far.

tl;dr - there are two types of Gen. Arbitrary for random values and edge cases and Progression for an exhaustive set of values (maybe progression could be renamed to Exhaustive?). Therefore, no need for an exhaustive parameter anymore. The distribution parameter is now part of the arbitrary constructors so each arbitrary is free to implement whatever distributions make sense for itself.

The arbitrary instances generate values that include a function for returning shrunk values. These shrunk values themselves then contain a function for further shrunk values and so on. This mechanism allows us to map and filter on gens and preserve the shrinking functionality.

The forAll method provides for minSuccess, maxFailure, a seed for the random instance, and a shrinking mode which can be used to disable shrinking, or bound the number of shrinks before stopping.

The iteration count for the property test is taken as the combination of the length of the sequences generated from the Gen instances). So for progressions, that’s the number of values in the closed set. For arbitraries you must pass in how many you want. This is better I believe because if you want to test 1000 random numbers in a function that also accepts a boolean, you can now have 1000 x true and 1000 x false, rather than 1000 x (random int, random boolean).

Code follows:

/**
 * A Generator, or [Gen] is responsible for generating data to be used in property testing.
 * Each generator will generate data for a specific type <T>.
 *
 * There are two supported types of generators - [Arbitrary] and [Progression] - defined
 * as sub interfaces of this interface.
 *
 * An arbitrary is used when you need random values across a large space.
 * A progression is useful when you want all values in a smaller closed space.
 *
 * Both types of generators can be mixed and matched in property tests. For example,
 * you could test a function with 1000 random positive integers (arbitrary) and every
 * even number from 0 to 200 (progression).
 */
interface Gen<T> {
   fun generate(random: Random): Sequence<PropertyInput<T>>
}

/**
 * An [Arbitrary] is a type of [Gen] which generates two types of values: edge cases and random values.
 *
 * Edge cases are values that should usually be included on every test run: those edge cases
 * which are common sources of bugs. For example, a function using ints is more likely to fail
 * for common edge cases like zero, minus 1, positive 1, [Int.MAX_VALUE] and [Int.MIN_VALUE]
 * rather than random values like 965489.
 *
 * The random values are used to give us a greater breadth to the test cases. In the case of a
 * function using ints, these random values could be from across the entire integer number line,
 * or could be limited to a subset of the integer space.
 */
interface Arbitrary<T> : Gen<T> {

   /**
    * Returns the values that are considered common edge case for the type.
    *
    * For example, for [String] this may include the empty string, a string with white space,
    * a string with unicode, and a string with non-printable characters.
    *
    * The result can be empty if for type T there are no common edge cases.
    *
    * @return the common edge cases for type T.
    */
   fun edgecases(): Iterable<T>

   /**
    * Returns a sequence of random [PropertyInput] values to be used for testing.
    *
    * @param random the [Random] instance to be used for generating values. This random instance is
    * seeded using the seed provided to the test framework so that tests can be deterministically re-run.
    * Implementations should honour the random provider whenever possible.
    *
    * @return the random test values as instances of [PropertyInput].
    */
   fun randoms(random: Random): Sequence<PropertyInput<T>>

   override fun generate(random: Random): Sequence<PropertyInput<T>> =
      edgecases().map { PropertyInput(it) }.asSequence() + randoms(random)

   companion object
}

fun <T, U> Arbitrary<T>.map(f: (T) -> U): Arbitrary<U> = object : Arbitrary<U> {
   override fun edgecases(): Iterable<U> = [e-mail]().map(f)
   override fun randoms(random: Random): Sequence<PropertyInput<U>> =
      [e-mail](random).map { it.map(f) }
}

fun <T> Arbitrary<T>.filter(predicate: (T) -> Boolean): Arbitrary<T> = object : Arbitrary<T> {
   override fun edgecases(): Iterable<T> = [e-mail]().filter(predicate)
   override fun randoms(random: Random): Sequence<PropertyInput<T>> =
      [e-mail](random).filter { predicate(it.value) }
}

/**
 * A [Progression] is a type of [Gen] which generates a deterministic, repeatable, and finite set
 * of values for a type T.
 *
 * An example of a progression is the range of integers from 0 to 100.
 * Another example is all strings of two characters.
 *
 * A progression is useful when you want to generate an exhaustive set of values from a given
 * sample space, rather than random values from that space. For example, if you were testing a
 * function that used an enum, you might prefer to guarantee that every enum value is used, rather
 * than selecting randomly from amongst the enum values (with possible duplicates and gaps).
 *
 * Progressions do not shrink their values. There is no need to find a smaller failing case, because
 * the smaller values will themselves naturally be included in the tested values.
 */
interface Progression<T> : Gen<T> {

   /**
    * @return the values for this progression as a lazy list.
    */
   fun values(): Sequence<T>

   override fun generate(random: Random): Sequence<PropertyInput<T>> = values().map { PropertyInput(it) }

   companion object
}

Some sample progressions:

fun Progression.Companion.int(range: IntRange) = object : Progression<Int> {
   override fun values(): Sequence<Int> = range.asSequence()
}

fun Progression.Companion.azstring(range: IntRange) = object : Progression<String> {
   private fun az() = ('a'..'z').asSequence().map { it.toString() }
   override fun values(): Sequence<String> = range.asSequence().flatMap { size ->
      List(size) { az() }.reduce { acc, seq -> acc.zip(seq).map { (a, b) -> a + b } }
   }
}

/**
 * Returns a [Progression] of the two possible boolean values - true and false.
 */
fun Progression.Companion.bools() = object : Progression<Boolean> {
   override fun values(): Sequence<Boolean> = sequenceOf(true, false)
}

Some sample arbitraries:

fun Arbitrary.Companion.int(
   iterations: Int,
   range: IntRange = Int.MIN_VALUE..Int.MAX_VALUE,
   distribution: IntDistribution = IntDistribution.Uniform
) = object : Arbitrary<Int> {
   override fun edgecases(): Iterable<Int> = listOf(Int.MIN_VALUE, Int.MAX_VALUE, 0)
   override fun randoms(random: Random): Sequence<PropertyInput<Int>> {
      return sequence {
         for (k in 0 until iterations) {
            val block = distribution.get(k, iterations, range.first.toLong()..range.last.toLong())
            val next = random.nextLong(block).toInt()
            val input = PropertyInput(next, IntShrinker)
            yield(input)
         }
      }
   }
}

sealed class IntDistribution {

   abstract fun get(k: Int, iterations: Int, range: LongRange): LongRange

   /**
    * Splits the range into discrete "blocks" to ensure that random values are distributed
    * across the entire range in a uniform manner.
    */
   object Uniform : IntDistribution() {
      override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
         val step = (range.last - range.first) / iterations
         return (step * k)..(step * (k + 1))
      }
   }

   /**
    * Values are distributed according to the Pareto distribution.
    * See https://en.wikipedia.org/wiki/Pareto_distribution
    * Sometimes referred to as the 80-20 rule
    *
    * tl;dr - more values are produced at the lower bound than the upper bound.
    */
   object Pareto : IntDistribution() {
      override fun get(k: Int, iterations: Int, range: LongRange): LongRange {
         // this isn't really the pareto distribution so either implement it properly, or rename this implementation
         val step = (range.last - range.first) / iterations
         return 0..(step * k + 1)
      }
   }
}

fun Arbitrary.Companion.long(
   iterations: Int,
   range: LongRange = Long.MIN_VALUE..Long.MAX_VALUE
) = object : Arbitrary<Long> {
   override fun edgecases(): Iterable<Long> = listOf(Long.MIN_VALUE, Long.MAX_VALUE, 0)
   override fun randoms(random: Random): Sequence<PropertyInput<Long>> {
      return sequence {
         for (k in 0 until iterations) {
            val next = random.nextLong(range)
            val input = PropertyInput(next, LongShrinker)
            yield(input)
         }
      }
   }
}

/**
 * Returns a stream of values where each value is a randomly
 * chosen Double.
 */
fun Arbitrary.Companion.double(): Arbitrary<Double> = object : Arbitrary<Double> {

   val literals = listOf(
      0.0,
      1.0,
      -1.0,
      1e300,
      Double.MIN_VALUE,
      Double.MAX_VALUE,
      Double.NEGATIVE_INFINITY,
      Double.NaN,
      Double.POSITIVE_INFINITY
   )

   override fun edgecases(): Iterable<Double> = literals

   override fun randoms(random: Random): Sequence<PropertyInput<Double>> {
      return generateSequence {
         val d = random.nextDouble()
         PropertyInput(d, DoubleShrinker)
      }
   }
}

fun Arbitrary.Companion.positiveDoubles(): Arbitrary<Double> = double().filter { it > 0.0 }
fun Arbitrary.Companion.negativeDoubles(): Arbitrary<Double> = double().filter { it < 0.0 }

/**
 * Returns an [Arbitrary] which is the same as [Arbitrary.double] but does not include +INFINITY, -INFINITY or NaN.
 *
 * This will only generate numbers ranging from [from] (inclusive) to [to] (inclusive)
 */
fun Arbitrary.Companion.numericDoubles(
   from: Double = Double.MIN_VALUE,
   to: Double = Double.MAX_VALUE
): Arbitrary<Double> = object : Arbitrary<Double> {
   val literals = listOf(0.0, 1.0, -1.0, 1e300, Double.MIN_VALUE, Double.MAX_VALUE).filter { it in (from..to) }
   override fun edgecases(): Iterable<Double> = literals
   override fun randoms(random: Random): Sequence<PropertyInput<Double>> {
      return generateSequence {
         val d = random.nextDouble()
         PropertyInput(d, DoubleShrinker)
      }
   }
}

And some example tests you might write:

fun main() {

   // tests 1000 random ints with all integers 0 to 100
   forAll(
      Arbitrary.int(1000),
      Progression.int(0..100)
   ) { a, b ->
      a + b == b + a
   }

   // tests 10 random longs in the range 11 to MaxLong with all combinations of a-z strings from 0 to 10 characters
   forAll(
      Arbitrary.long(10, 11..Long.MAX_VALUE),
      Progression.azstring(0..10)
   ) { a, b ->
      b.length < a
   }

   // convenience functions which mimics the existing style
   // tests random longs, using reflection to pick up the Arbitrary instances
   // each arb will have the same number of iterations
   forAll<Long, Long>(1000) { a, b -> a + b == b + a }
}

You can see all this in the io.kotest.property package in module kotest-assertions.

[xgouchet]

@Kerooker regarding the deterministic re runs, I think what should be done is by default leave the seed null (changing everytime), and if a test failed, print to System.err the seed that was used for the failing test.

You can then apply that same seed to reproduce the failing case, fix your code so the tests pass, and then remove the seed again to resume using random seeds.