concurrentTests not being respected during ConcurrentTestLauncher [kotest 4.6.3]

During some experimenting with kotest, I was playing around with the parallelism, concurrentTests and concurrentSpecs.

When setting the concurrentTests to something higher than 1, it just burst and execute all the tests all at once, even if I have a delay (which could be due to being suspended) or if I have hacky stuff like doing nothing for X amount of seconds (to ensure no suspension points were available) and it still burst the number of tests being triggered all at once.

I noticed that it relies on kotlinx.coroutines.sync.Semaphore and on Semaphore.withPermit.

Also, what I should expect if I have 4 Specs with 12 tests each and I set the values to concurrentTests = 3 and concurrentSpecs = 2?

As far as I understood the code, 1 coroutine should be launched for each spec and 3 coroutines should be launched for each test, totaling 6 concurrent tests running at once.

If that’s the case, seems that concurrentTest is not super useful, since I can’t have fine tunning on how many tests I’m running all at once. I don’t whether this makes sense or not to be honest, but if I set the concurrentTests for 10, I understand that I shouldn’t have more than 10 tests running at the same time globally.

One solution could be this also be configurable, like IsolationMode, so I could configure concurrentTests per Spec or Globally.

Kotest version: 4.6.3 Java version: Adopt 10.0.11-hs-adpt Kotlin version: 1.5.31

[sksamuel]

I think there’s a coroutine scope in the loop too to stop that happening. I’ll check more and give a full reply to your posts tomorrow night when I’m back from vacation.

On Sat, 6 Nov 2021, 16:10 William Johnson S. Okano, < @.***> wrote:

I’ve found the problem. On version 4.6.3, ConcurrentTestLauncher.launch is using the semaphore wrong.

Instead of using semaphore.withPermit inside the launched coroutine, it’s using inside the for loop. As coroutines launched by launch as fire-and-forget, the permit is being acquired and released even if the coroutines didn’t execute.

The implementation on DefaultSpecLauncher doesn’t contain that issue and that’s why the value is respected.

I would open a PR to fix this, but I took a quick look and it seems that on the 5.0.0 the structure changed quite a bit, and I still don’t know whether the problem persists.

Should I crate a PR for 4.6.3 branch in order to fix this issue or should I leave it as it is?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kotest/kotest/issues/2623#issuecomment-962473288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFVSGST4PBISTMS3IXDZKDUKVHNZANCNFSM5HN4PTNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[sksamuel]

Now that 5.0 is out and you’re happy with the current state, I will close this.