Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Deadlock after upgrading to Mockito 2 and parallel tests execution

See original GitHub issue

I updated Mockito and ScalaTest today from ScalaTest 2.x and Mockito 1.x and my tests suddenly froze in SBT (not in IntelliJ). After some playing around, I noticed every test class by itself passes and when I disabled running tests in parallel in SBT, the tests would work again.

I used jstack and came across this deadlock:

Found one Java-level deadlock:
=============================
"pool-9-thread-10-ScalaTest-running-ApiKeyVerificationFilterSpec":
  waiting to lock monitor 0x00007fea6f039538 (object 0x00000007959e4a78, a sbt.classpath.ClasspathUtilities$$anon$1),
  which is held by "pool-9-thread-2-ScalaTest-running-BRequestFilterSpec"
"pool-9-thread-2-ScalaTest-running-BRequestFilterSpec":
  waiting to lock monitor 0x00007fea70e95548 (object 0x0000000795a868e8, a sbt.classpath.ClasspathFilter),
  which is held by "pool-9-thread-10-ScalaTest-running-ApiKeyVerificationFilterSpec"

Java stack information for the threads listed above:
===================================================
"pool-9-thread-10-ScalaTest-running-ApiKeyVerificationFilterSpec":
	at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
	- waiting to lock <0x00000007959e4a78> (a sbt.classpath.ClasspathUtilities$$anon$1)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	- locked <0x0000000795a868e8> (a sbt.classpath.ClasspathFilter)
	at sbt.classpath.ClasspathFilter.loadClass(ClassLoaders.scala:59)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at net.bytebuddy.dynamic.loading.MultipleParentClassLoader.loadClass(MultipleParentClassLoader.java:68)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at sun.reflect.GeneratedSerializationConstructorAccessor35.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:48)
	at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:73)
	at org.mockito.internal.creation.instance.ObjenesisInstantiator.newInstance(ObjenesisInstantiator.java:18)
	at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:47)
	at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
	at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
	at org.mockito.Mockito.mock(Mockito.java:1729)
	at org.mockito.Mockito.mock(Mockito.java:1642)
	at org.scalatest.mockito.MockitoSugar$class.mock(MockitoSugar.scala:73)
	at com.b.n.filter.ApiKeyVerificationFilterSpec.mock(ApiKeyVerificationFilterSpec.scala:25)
	at com.b.n.filter.ApiKeyVerificationFilterSpec$ApiKeyFetchSetup$class.$init$(ApiKeyVerificationFilterSpec.scala:77)
	at com.b.n.filter.ApiKeyVerificationFilterSpec$$anonfun$15$$anon$11.<init>(ApiKeyVerificationFilterSpec.scala:80)
	at com.b.n.filter.ApiKeyVerificationFilterSpec$$anonfun$15.apply(ApiKeyVerificationFilterSpec.scala:80)
	at com.b.n.filter.ApiKeyVerificationFilterSpec$$anonfun$15.apply(ApiKeyVerificationFilterSpec.scala:80)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSpecLike$$anon$1.apply(FunSpecLike.scala:454)
	at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
	at org.scalatest.FunSpec.withFixture(FunSpec.scala:1630)
	at org.scalatest.FunSpecLike$class.invokeWithFixture$1(FunSpecLike.scala:451)
	at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
	at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
	at org.scalatest.FunSpecLike$class.runTest(FunSpecLike.scala:464)
	at com.b.n.filter.ApiKeyVerificationFilterSpec.org$scalatest$BeforeAndAfter$$super$runTest(ApiKeyVerificationFilterSpec.scala:25)
	at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:203)
	at com.b.n.filter.ApiKeyVerificationFilterSpec.runTest(ApiKeyVerificationFilterSpec.scala:25)
	at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
	at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
	at org.scalatest.FunSpecLike$class.runTests(FunSpecLike.scala:497)
	at org.scalatest.FunSpec.runTests(FunSpec.scala:1630)
	at org.scalatest.Suite$class.run(Suite.scala:1147)
	at org.scalatest.FunSpec.org$scalatest$FunSpecLike$$super$run(FunSpec.scala:1630)
	at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
	at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
	at org.scalatest.FunSpecLike$class.run(FunSpecLike.scala:501)
	at com.b.n.filter.ApiKeyVerificationFilterSpec.org$scalatest$BeforeAndAfter$$super$run(ApiKeyVerificationFilterSpec.scala:25)
	at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:258)
	at com.b.n.filter.ApiKeyVerificationFilterSpec.run(ApiKeyVerificationFilterSpec.scala:25)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
	at sbt.TestRunner.runTest$1(TestFramework.scala:76)
	at sbt.TestRunner.run(TestFramework.scala:85)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
	at sbt.TestFunction.apply(TestFramework.scala:207)
	at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
	at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"pool-9-thread-2-ScalaTest-running-BRequestFilterSpec":
	at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
	- waiting to lock <0x0000000795a868e8> (a sbt.classpath.ClasspathFilter)
	at sbt.classpath.ClasspathFilter.loadClass(ClassLoaders.scala:59)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at net.bytebuddy.dynamic.loading.MultipleParentClassLoader.loadClass(MultipleParentClassLoader.java:68)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$Direct.defineClass(ClassInjector.java:408)
	at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection.inject(ClassInjector.java:185)
	- locked <0x0000000781ee0ff0> (a net.bytebuddy.dynamic.loading.MultipleParentClassLoader)
	at net.bytebuddy.dynamic.loading.ClassLoadingStrategy$Default$InjectionDispatcher.load(ClassLoadingStrategy.java:187)
	at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:79)
	at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:4376)
	at org.mockito.internal.creation.bytebuddy.SubclassBytecodeGenerator.mockClass(SubclassBytecodeGenerator.java:94)
	at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:37)
	at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:34)
	at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:138)
	at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:346)
	at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:161)
	- locked <0x00000007959e4a78> (a sbt.classpath.ClasspathUtilities$$anon$1)
	at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:355)
	at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:32)
	at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMockType(SubclassByteBuddyMockMaker.java:71)
	at org.mockito.internal.creation.bytebuddy.SubclassByteBuddyMockMaker.createMock(SubclassByteBuddyMockMaker.java:42)
	at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:25)
	at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:63)
	at org.mockito.Mockito.mock(Mockito.java:1729)
	at org.mockito.Mockito.mock(Mockito.java:1642)
	at org.scalatest.mockito.MockitoSugar$class.mock(MockitoSugar.scala:73)
	at com.b.n.filter.BRequestFilterSpec.mock(BRequestFilterSpec.scala:18)
	at com.b.n.filter.BRequestFilterSpec$Base$class.$init$(BRequestFilterSpec.scala:31)
	at com.b.n.filter.BRequestFilterSpec$$anonfun$7$$anon$10.<init>(BRequestFilterSpec.scala:43)
	at com.b.n.filter.BRequestFilterSpec$$anonfun$7.apply(BRequestFilterSpec.scala:43)
	at com.b.n.filter.BRequestFilterSpec$$anonfun$7.apply(BRequestFilterSpec.scala:43)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSpecLike$$anon$1.apply(FunSpecLike.scala:454)
	at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
	at org.scalatest.FunSpec.withFixture(FunSpec.scala:1630)
	at org.scalatest.FunSpecLike$class.invokeWithFixture$1(FunSpecLike.scala:451)
	at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
	at org.scalatest.FunSpecLike$$anonfun$runTest$1.apply(FunSpecLike.scala:464)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
	at org.scalatest.FunSpecLike$class.runTest(FunSpecLike.scala:464)
	at com.b.n.filter.BRequestFilterSpec.org$scalatest$BeforeAndAfter$$super$runTest(BRequestFilterSpec.scala:18)
	at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:203)
	at com.b.n.filter.BRequestFilterSpec.runTest(BRequestFilterSpec.scala:18)
	at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
	at org.scalatest.FunSpecLike$$anonfun$runTests$1.apply(FunSpecLike.scala:497)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
	at org.scalatest.FunSpecLike$class.runTests(FunSpecLike.scala:497)
	at org.scalatest.FunSpec.runTests(FunSpec.scala:1630)
	at org.scalatest.Suite$class.run(Suite.scala:1147)
	at org.scalatest.FunSpec.org$scalatest$FunSpecLike$$super$run(FunSpec.scala:1630)
	at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
	at org.scalatest.FunSpecLike$$anonfun$run$1.apply(FunSpecLike.scala:501)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
	at org.scalatest.FunSpecLike$class.run(FunSpecLike.scala:501)
	at com.b.n.filter.BRequestFilterSpec.org$scalatest$BeforeAndAfter$$super$run(BRequestFilterSpec.scala:18)
	at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:258)
	at com.b.n.filter.BRequestFilterSpec.run(BRequestFilterSpec.scala:18)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
	at sbt.TestRunner.runTest$1(TestFramework.scala:76)
	at sbt.TestRunner.run(TestFramework.scala:85)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
	at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
	at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
	at sbt.TestFunction.apply(TestFramework.scala:207)
	at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
	at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

ApiKeyVerificationFilter, line 77 is val apikey = mock[ImmutableMap[String, ChildData]] and BRequestFilterSpec line 31 is val pmap = mock[ParamMap].

sbt prompt looks like this:

$ sbt8             
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
[info] Loading global plugins from /Users/m/.sbt/0.13/plugins
[info] Loading project definition from  ...
[info] Set current project to newton-api (in build file:...)
> test
[info] NServiceModuleSpec:
[info] NServiceModule
[info] - should have a test !!! IGNORED !!!

-- hangs here --

Mac OS 10.11.6 java version “1.8.0_45” Java™ SE Runtime Environment (build 1.8.0_45-b14) Java HotSpot™ 64-Bit Server VM (build 25.45-b02, mixed mode) sbt 0.13.9 “org.scalatest” %% “scalatest” % “3.0.1” % “test”, “org.mockito” % “mockito-core” % “2.7.22” % “test”

parallelExecution in Test := true in build.sbt - setting this to false circumvents the issue.