Lots of Builds in the community-build are silently skipping tests and even test-compiles

It looks like lots of Builds not running or even compiling their tests. When they get to testing this happens:

2022-01-30T08:16:22.4896099Z [error] Error loading test framework (java.lang.NoClassDefFoundError: sbt/testing/Framework). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set
2022-01-30T08:16:22.4899342Z [error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.ScalaLibrary
2022-01-30T08:16:22.4900029Z [error] or
2022-01-30T08:16:22.4900718Z [error] Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
2022-01-30T08:16:22.5477827Z [success] Total time: 23 s, completed Jan 30, 2022 9:16:22 AM

As similar issues happens for some builds trying to use ZTestFramework:

2022-01-30T08:21:26.8954542Z [error] Error loading test framework (scala.MatchError: zio.test.sbt.ZTestFramework@31fd0360 (of class zio.test.sbt.ZTestFramework)). This usually means that you are using a layered class loader that cannot reach the sbt.testing.Framework class. The most likely cause is that your project has a runtime dependency on your test framework, e.g. scalatest. To fix this, you can try to set

When this error occurs in a project, none of the tests will be run or even compiled. It is effectively a total disabling of testing code for the given project.

I first saw these in the my quill PR builds:

Some quick grepping (attached) shows me this is happening for the following projects:

in community-build-a

• scalatest
• scala-stm
• scalatestplus-testng
• izumi-reflect
• Lucre

in community-build-c

• protoquill (I got this one…)
• akka
• jackson-module-scala
• libretto
• xml-interpolator
• scalatestplus-scalacheck
• scala-parallel-collections
• sconfig
• onnx-scala

I solved this problem by adding something slightly different to my settings:

Compile / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat

When I added this, the error went away.

This could all be totally wrong and my grepping could be incorrect too but I’ve tried this with the quill build. I tried running the community build with and actual compile error in my scalatests and without the classLoaderLayeringStrategy parameter. The build passes as though nothing is wrong.

I think that project owners should maybe have a look at their builds and see if this is happening for them.

grepping_results.txt

[eed3si9n]

I have a PR up so it won’t silently succeed the build - https://github.com/sbt/sbt/pull/6806

[griggt]

our community build apparently still injects the obsolete sbt-dotty plugin

Any objection to discontinuing this practice and manually fixing up in the dotty-staging forks any projects that have not updated upstream?