java.io.IOException: Resource deadlock would occur

sbt version: 1.3.3

steps

Running parallel build (2 different jvm) in CI with sbt compile/test in 2 different projects

problem

Sbt fail with a resource deadlock on coursier cache (not often… difficult to reproduce)

expectation

It should be possible to run two concurrent build without having coursier deadlock

notes

stack trace

Executing command:sbt -Dprotobufnoembedded clean ${{SBT_DOCKER}} < /dev/null
[info] Loading settings for project dispatch-service-build from plugins.sbt,scalapb.sbt ...
[info] Loading project definition from /codefresh/volume/dispatch-service/project
[info] Loading settings for project dispatch-service from build.sbt ...   
[success] Total time: 1 s, completed Nov 22, 2019 12:00:56 PM
[info] Formatting 74 Scala sources...
[info] Formatting 32 Scala sources...
[info] Formatting 1 Scala sources...
[info] Formatting 31 Scala sources...
[info] Formatting 66 Scala sources...
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 31 Scala sources to /codefresh/volume/dispatch-service/plans/target/scala-2.12/classes ...
[info] Compiling 32 Scala sources to /codefresh/volume/dispatch-service/api/target/scala-2.12/classes ...
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6f8c396 rejected from java.util.concurrent.ThreadPoolExecutor@83f3d39[Shutting down, pool size = 5, active threads = 5, queued tasks = 0, completed tasks = 7797]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at scala.concurrent.Promise.complete(Promise.scala:53)
	at scala.concurrent.Promise.complete$(Promise.scala:52)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:187)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@52b1c4b6 rejected from java.util.concurrent.ThreadPoolExecutor@83f3d39[Shutting down, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 7798]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at scala.concurrent.Promise.complete(Promise.scala:53)
	at scala.concurrent.Promise.complete$(Promise.scala:52)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:187)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@6e73c92e rejected from java.util.concurrent.ThreadPoolExecutor@83f3d39[Shutting down, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 7799]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at scala.concurrent.Promise.complete(Promise.scala:53)
	at scala.concurrent.Promise.complete$(Promise.scala:52)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:187)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@38a201c5 rejected from java.util.concurrent.ThreadPoolExecutor@83f3d39[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 7800]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at scala.concurrent.Promise.complete(Promise.scala:53)
	at scala.concurrent.Promise.complete$(Promise.scala:52)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:187)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[info] Non-compiled module 'compiler-bridge_2.12' for Scala 2.12.8. Compiling...
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable@516baabf rejected from java.util.concurrent.ThreadPoolExecutor@83f3d39[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 7801]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
	at scala.concurrent.Promise.complete(Promise.scala:53)
	at scala.concurrent.Promise.complete$(Promise.scala:52)
	at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:187)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[info]   Compilation completed in 23.519s.
Waiting for lock on /codefresh/volume/caches/sbt/boot/scala-2.12.10/org.scala-sbt/sbt/1.3.3/sbt.components.lock to be available...
[info] Done compiling.
[info] Done compiling.
[info] Compiling 66 Scala sources to /codefresh/volume/dispatch-service/domain/target/scala-2.12/classes ...
[info] Done compiling.
[error] java.io.IOException: Resource deadlock would occur
[error] 	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
[error] 	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:94)
[error] 	at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1072)
[error] 	at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
[error] 	at lmcoursier.internal.shaded.coursier.paths.CachePath.withStructureLock(CachePath.java:132)
[error] 	at lmcoursier.internal.shaded.coursier.cache.CacheLocks$.withStructureLock(CacheLocks.scala:21)
[error] 	at lmcoursier.internal.shaded.coursier.cache.CacheLocks$.withLockOr(CacheLocks.scala:38)
[error] 	at lmcoursier.internal.shaded.coursier.cache.FileCache.$anonfun$download$32(FileCache.scala:553)
[error] 	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error] 	at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error] 	at scala.util.Success.map(Try.scala:213)
[error] 	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error] 	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error] 	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] 	at java.lang.Thread.run(Thread.java:748)
[error] (infrastructure / update) java.io.IOException: Resource deadlock would occur
[error] Total time: 195 s (03:15), completed Nov 22, 2019 12:04:12 PM

Difficult to reproduce, i have this in only one project for the moment, if you have some clue how i can reduce this to a minimal failing case, help is welcomed

[eed3si9n]

I noticed it says:

Waiting for lock on /codefresh/volume/caches/sbt/boot/scala-2.12.10/org.scala-sbt/sbt/1.3.3/sbt.components.lock to be available...

That’s likely from sbt launcher

• https://github.com/sbt/launcher/blob/fbe82523cdfe5453642f1dccbd2415cc806093a1/launcher-implementation/src/main/scala/xsbt/boot/Launch.scala#L256
• https://github.com/sbt/launcher/blob/1.x/launcher-implementation/src/main/scala/xsbt/boot/Locks.scala#L90

So at some point in time two instances of sbt would end up holding on to sbt launcher’s lock or Coursier’s lock each, both waiting on each other? Does that scenario make sense?

[gmarti]

Anything new here ? @alexarchambault