Extension function forEachBatch can be added for DataStreamWriter

I found it difficult to call DataStreamWriter.foreachBatch because source code won’t compile until explicit construction of VoidFunction2 is added.

So I suggest adding such an extension for DataStreamWriter:

import org.apache.spark.api.java.function.VoidFunction2
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.streaming.DataStreamWriter

public fun <T> DataStreamWriter<T>.forEachBatch(
    func: (batch: Dataset<T>, batchId: Long) -> Unit
): DataStreamWriter<T> = foreachBatch(
    VoidFunction2 { batch, batchId ->
        func(batch, batchId)
    }
)

[Pihanya]

@hawkaa, hello! Thank you for the remark.

In Kotlin we follow coding conventions. It is said there: “Names of functions, properties and local variables start with a lowercase letter and use camel case”. This is the first argument why forEachBatch is better option vs foreachBatch.

Also in kotlin-spark-api we have forEach function that calls Spark’s foreach under the hood. So forEachBatch would be more idiomatic (in terms of kotlin-spark-api) comparing to Spark’s foreachBatch.

Therefore, if choosing function naming between forEachBatch vs foreachBatch for kotlin-spark-api it is more propriate to use the first variant, that is available in release v1.1.0.

[Jolanrensen]

Your specific example doesn’t work because List<CatalogueNode> cannot be encoded. It’s an interface which can have functions, values etc. so Spark does not know how to encode that. Only a collection of actual data classes would be allowed. So like

val folderChildren: List<Folder>? = null,
val identityRefChildren: List<IdentityRef>? = null,

But then again, the circular reference appears of course. Unfortunately, I don’t think we have a solution for that. Especially since Spark itself does not support circular references. It makes sense if you consider that Datasets are essentially column/row data structures. If circular references were allowed, an infinite recursion could exist within a cell which cannot be saved. Some things I found regarding this: https://issues.apache.org/jira/browse/SPARK-33598

As for java.util.UUID, Spark does not support this as well, so I think it’s outside the scope of the Kotlin Spark API to add support for this specific class. Usually we only mirror org.apache.spark.sql.Encoders