Support for snapshot testing

Consider the following test case:

Given a SQL query the system should produce the following results (in any order). The case is repeated for various SQL queries.

Of course I can use

sql = "..."
assertEquals("results for ${sql}", """
   expected line 1
   expected line 2
""",  execute(sql).sorted())
sql = "..."
assertEquals("results for ${sql}", """
   expected line 1
   expected line 2
""",  execute(sql).sorted())
sql = "..."
assertEquals("results for ${sql}", """
   expected line 1
   expected line 2
""",  execute(sql).sorted())

however if the implementation changes, then it would be very hard to update all the test to new expected values. It would basically force the developer to manually go and update each and every assert statement.

I wonder if kotlintest could integrate something like “self-reproducing” test cases, so it creates “new test code” as one of the results of the test execution.

That is:

1. It performs the assertions as usual
2. It writes “target/tmp/…kt” file with new source code. If developer agrees that all the “failures” are expected, he can just grab the new file and copy it over the existing one

Here’s an example of such a test: https://github.com/apache/calcite/blob/3c6b5ec759caadabb67f09d7a4963cc7d9386d0c/core/src/test/resources/sql/join.iq

For instance, !plan at line 62 asserts that execution plan of a previous query matches the plan listed at lines 39…61. The result of .iq execution is a brand new .iq file with “expected values” set to the current actuals.

In other words, if a test passes, then .iq is a quine.

.iq stands for https://github.com/julianhyde/quidem there.

I wonder if that test idiom can be incorporated and expressed in Kotlin.

Pros are as follows:

1. The test is somewhat easy to read
2. The test is easy to write: just start with EMPTY assertion, and test output would produce the text of the test WITH expected value for you
3. It is trivial to maintain: one reviews the diff, then copies new file over the old one
4. It is fun

[sksamuel]

Will look to include this for 5.1

[sidgate]

Approval tests works well for API testing, I can just have the JSON verified. Now I cannot use kotest for such scenarios.