Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Workflow fails with slick error with large scale call cache

See original GitHub issue

Tried to re-run the 10K JG workflow with CC on, the workflow failed almost immediately with multiple errors like

[ERROR] [04/18/2017 21:11:44.685] [cromwell-system-akka.dispatchers.service-dispatcher-86] [akka://cromwell-system/user/cromwell-service/ServiceRegistryActor/MetadataService/metadata-summary-actor] Failed to summarize metadata
java.util.concurrent.RejectedExecutionException: Task slick.basic.BasicBackend$DatabaseDef$$anon$2@64919660 rejected from slick.util.AsyncExecutor$$anon$2$$anon$1@1dce40e4[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 800]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
	at slick.util.AsyncExecutor$$anon$2$$anon$3.execute(AsyncExecutor.scala:120)
	at slick.basic.BasicBackend$DatabaseDef$class.runSynchronousDatabaseAction(BasicBackend.scala:233)
	at slick.jdbc.JdbcBackend$DatabaseDef.runSynchronousDatabaseAction(JdbcBackend.scala:38)
	at slick.basic.BasicBackend$DatabaseDef$class.runInContext(BasicBackend.scala:210)
	at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
	at slick.basic.BasicBackend$DatabaseDef$class.run$1(BasicBackend.scala:153)
	at slick.basic.BasicBackend$DatabaseDef$class.runInContext(BasicBackend.scala:157)
	at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
	at slick.basic.BasicBackend$DatabaseDef$class.runInContext(BasicBackend.scala:179)
	at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
	at slick.basic.BasicBackend$DatabaseDef$class.runInternal(BasicBackend.scala:78)
	at slick.jdbc.JdbcBackend$DatabaseDef.runInternal(JdbcBackend.scala:38)
	at slick.basic.BasicBackend$DatabaseDef$class.run(BasicBackend.scala:75)
	at slick.jdbc.JdbcBackend$DatabaseDef.run(JdbcBackend.scala:38)
	at cromwell.database.slick.SlickDatabase.runTransaction(SlickDatabase.scala:103)
	at cromwell.database.slick.MetadataSlickDatabase$class.refreshMetadataSummaryEntries(MetadataSlickDatabase.scala:133)
	at cromwell.database.slick.SlickDatabase.refreshMetadataSummaryEntries(SlickDatabase.scala:54)
	at cromwell.services.metadata.impl.MetadataDatabaseAccess$class.refreshWorkflowMetadataSummaries(MetadataDatabaseAccess.scala:146)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor.refreshWorkflowMetadataSummaries(MetadataSummaryRefreshActor.scala:35)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor$$anonfun$1.applyOrElse(MetadataSummaryRefreshActor.scala:45)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor$$anonfun$1.applyOrElse(MetadataSummaryRefreshActor.scala:43)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
	at akka.actor.FSM$class.processEvent(FSM.scala:663)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor.akka$actor$LoggingFSM$$super$processEvent(MetadataSummaryRefreshActor.scala:35)
	at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor.processEvent(MetadataSummaryRefreshActor.scala:35)
	at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
	at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
	at cromwell.services.metadata.impl.MetadataSummaryRefreshActor.aroundReceive(MetadataSummaryRefreshActor.scala:35)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

Issue Analytics

State:
Created 6 years ago
Comments:12 (8 by maintainers)

Top GitHub Comments

1reaction

geoffjentrycommented, Apr 26, 2017

Is upping queueSize the answer here?

0reactions

geoffjentrycommented, Aug 18, 2018

Since @mcovarr and @Horneth are currently looking at call caching, please make sure that this no longer is an issue

Top Results From Across the Web

CallCaching - Cromwell - Read the Docs

Call Caching allows Cromwell to detect when a job has been run in the past so that it doesn't have to re-compute results,...

Github Actions for Elixir CI - Fly.io

If our caches ever cause a problem, and sometimes they can, it's good to know how to clear them. In our project, go...

Changelogs | dotCMS

Bad results on ShortyIdentifier database calls are no longer cached. This prevents a condition in which a database error could result in ...

SMRT® Link Troubleshooting Guide (v8.0) - PacBio

call -caching { enabled = true invalidate-bad-cache-results = true. } workflow-options { workflow-log-temporary = false. } database { profile = "slick.jdbc.

Chapter 16 Storage | The drake R Package User Manual

16.2 Efficient target storage. drake supports custom formats for large and specialized targets. For example, the "fst" format uses the fst package to...