combine(Iterable<Flow>) is very slow

The combine(…) function becomes very slow the depending on the amount of Flows.

Flat-mapping a 40k List<List<…>> takes approx. 11ms. With one element per list. Flat-mapping a 40k List<Flow<…>> takes approx. 1 minute. With one element per Flow. And that’s with a very very simple non-suspending test case.

If combine is expected to have such a low performance it should be documented, otherwise developers like me treat is as a simple Flow-version of List<List<…>>.flatten(). If it’s not expected then there is a bug.

My use case

I have ~40k objects in memory. For each object there is a Flow and a coroutine that periodically refreshes the object from the server when it’s expired and emits that. At some point I have to continuously merge the latest version of all 40k objects into a single list for further processing. For that I use combine(listOfObjectFlows) { it.toList() }.

Unfortunately that takes somewhere north of 15 minutes already. I have no time to let it finish to see the total…

I’ve written my own implementation now as a workaround.

Test code

import kotlin.time.*
import kotlinx.coroutines.flow.*

@OptIn(ExperimentalTime::class)
suspend fun main() {
    val flowList = (1..40_000).map { flowOf(it) }
    val start = TimeSource.Monotonic.markNow()
    val listFlow = combine(flowList) { it.toList() }

    listFlow.collect {
        println("Took: ${start.elapsedNow()}") // Took: 66.5s
    }
}

[qwwdfsad]

Even with a 40% gain the timing would still be a different ballpark than what I expect (minutes instead of seconds).

The problem was pretty simple – accidental and well-hidden O(N^2) asymptotic where N is the number of flows. I’ve fixed it, it will be available in 1.4.0 and will be even faster (by a significant margin) than proposed implementation. In general, for a large number of flows, combine became faster by orders of magnitude and for two flows (“basic case”) it became precisely two times faster.

Thanks for pointing it out!

[qwwdfsad]

Yes, with suspend fun main there is no dispatcher, so multithreaded Dispatchers.Default is used and adds non-determinism. I was implying deterministic case when combine is called from within a runBlocking or Dispatchers.Main