Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

enable parsing of packed cwl containing `SchemaDefRequirement` with multiple `$import:`

See original GitHub issue

I am attempting to run a packed CWL workflow.

The output from the latest cromwell is:

(p3cwl) [jeremiah@localhost ~]$ git clone https://github.com/NCI-GDC/gdc-dnaseq-cwl.git
(p3cwl) [jeremiah@localhost ~]$ java -jar cromwell/server/target/scala-2.12/cromwell-35-fd560e9-SNAP.jar run gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl --inputs gdc-dnaseq-cwl/workflows/bamfastq_align/transform_c9bab596-0b03-4066-b428-d9cd36fb1c.yml --type CWL --workflow-root main
[2018-09-14 13:19:10,55] [info] Running with database db.url = jdbc:hsqldb:mem:d07a09a8-8d20-4095-967b-c6f375a3f309;shutdown=false;hsqldb.tx=mvcc
[2018-09-14 13:19:19,53] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2018-09-14 13:19:19,55] [info] [RenameWorkflowOptionsInMetadata] 100%
[2018-09-14 13:19:19,67] [info] Running with database db.url = jdbc:hsqldb:mem:e41fe9de-508c-4f49-aeaa-ce7474d7c1e2;shutdown=false;hsqldb.tx=mvcc
[2018-09-14 13:19:20,18] [info] Slf4jLogger started
[2018-09-14 13:19:20,25] [info] Pre Processing Workflow...
[2018-09-14 13:19:20,65] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl
[2018-09-14 13:19:54,70] [info] Pre Processing Inputs...
[2018-09-14 13:19:54,94] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-89ab52b",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2018-09-14 13:19:55,04] [info] Metadata summary refreshing every 2 seconds.
[2018-09-14 13:19:55,31] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-14 13:19:55,32] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-14 13:19:55,32] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-09-14 13:19:56,83] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-09-14 13:19:56,88] [info] SingleWorkflowRunnerActor: Version 35-fd560e9-SNAP
[2018-09-14 13:19:56,91] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-09-14 13:19:57,89] [info] CWL (Unspecified version) workflow caab4283-a3d4-4966-85ba-56d0992c8f00 submitted
[2018-09-14 13:19:57,90] [info] SingleWorkflowRunnerActor: Workflow submitted caab4283-a3d4-4966-85ba-56d0992c8f00
[2018-09-14 13:19:57,91] [info] 1 new workflows fetched
[2018-09-14 13:19:57,92] [info] WorkflowManagerActor Starting workflow caab4283-a3d4-4966-85ba-56d0992c8f00
[2018-09-14 13:19:57,93] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-09-14 13:19:57,96] [info] WorkflowManagerActor Successfully started WorkflowActor-caab4283-a3d4-4966-85ba-56d0992c8f00
[2018-09-14 13:19:57,96] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-09-14 13:19:57,96] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-09-14 13:19:58,11] [info] MaterializeWorkflowDescriptorActor [caab4283]: Parsing workflow as CWL v1.0
[2018-09-14 13:20:00,08] [error] WorkflowManagerActor Workflow caab4283-a3d4-4966-85ba-56d0992c8f00 failed (during MaterializingWorkflowDescriptorState): cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anon$1: Workflow input processing failed:
Custom type file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_fastq_pe_file was referred to but not found in schema def Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#amplicon_kit.yml/amplicon_kit_set_file,Some([Lcwl.InputRecordField;@5f7affaf),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#amplicon_kit.yml/amplicon_kit_set_uuid,Some([Lcwl.InputRecordField;@4e2399d2),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#capture_kit.yml/capture_kit_set_file,Some([Lcwl.InputRecordField;@74284741),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#capture_kit.yml/capture_kit_set_uuid,Some([Lcwl.InputRecordField;@7f378344),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_meta,Some([Lcwl.InputRecordField;@77389c8a),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_fastq_pe_file,Some([Lcwl.InputRecordField;@258776b6),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_fastq_se_file,Some([Lcwl.InputRecordField;@13a32f08),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroups_bam_file,Some([Lcwl.InputRecordField;@50cda39b),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_fastq_pe_uuid,Some([Lcwl.InputRecordField;@63a2b9c4),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroup_fastq_se_uuid,Some([Lcwl.InputRecordField;@7d8e51c0),record,None)), Inl(InputRecordSchema(file:///home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform_pack.cwl#readgroup.yml/readgroups_bam_uuid,Some([Lcwl.InputRecordField;@775ffa16),record,None)).
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.cromwell$engine$workflow$lifecycle$materialization$MaterializeWorkflowDescriptorActor$$workflowInitializationFailed(MaterializeWorkflowDescriptorActor.scala:211)
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anonfun$2.applyOrElse(MaterializeWorkflowDescriptorActor.scala:181)
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor$$anonfun$2.applyOrElse(MaterializeWorkflowDescriptorActor.scala:176)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34)
	at akka.actor.FSM.processEvent(FSM.scala:684)
	at akka.actor.FSM.processEvent$(FSM.scala:681)
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.akka$actor$LoggingFSM$$super$processEvent(MaterializeWorkflowDescriptorActor.scala:135)
	at akka.actor.LoggingFSM.processEvent(FSM.scala:820)
	at akka.actor.LoggingFSM.processEvent$(FSM.scala:802)
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.processEvent(MaterializeWorkflowDescriptorActor.scala:135)
	at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:678)
	at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:672)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.aroundReceive(MaterializeWorkflowDescriptorActor.scala:135)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2018-09-14 13:20:00,08] [info] WorkflowManagerActor WorkflowActor-caab4283-a3d4-4966-85ba-56d0992c8f00 is in a terminal state: WorkflowFailedState
[2018-09-14 13:20:00,92] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-09-14 13:20:05,34] [info] Workflow polling stopped
[2018-09-14 13:20:05,36] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-09-14 13:20:05,36] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-09-14 13:20:05,37] [info] Aborting all running workflows.
[2018-09-14 13:20:05,39] [info] WorkflowStoreActor stopped
[2018-09-14 13:20:05,36] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-09-14 13:20:05,39] [info] JobExecutionTokenDispenser stopped
[2018-09-14 13:20:05,40] [info] WorkflowLogCopyRouter stopped
[2018-09-14 13:20:05,40] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-09-14 13:20:05,40] [info] WorkflowManagerActor All workflows finished
[2018-09-14 13:20:05,40] [info] WorkflowManagerActor stopped
[2018-09-14 13:20:05,40] [info] Connection pools shut down
[2018-09-14 13:20:05,41] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-09-14 13:20:05,41] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-09-14 13:20:05,41] [info] SubWorkflowStoreActor stopped
[2018-09-14 13:20:05,41] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-09-14 13:20:05,41] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-09-14 13:20:05,42] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-09-14 13:20:05,42] [info] JobStoreActor stopped
[2018-09-14 13:20:05,42] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-09-14 13:20:05,42] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-09-14 13:20:05,42] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-09-14 13:20:05,43] [info] CallCacheWriteActor stopped
[2018-09-14 13:20:05,43] [info] DockerHashActor stopped
[2018-09-14 13:20:05,43] [info] IoProxy stopped
[2018-09-14 13:20:05,43] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-09-14 13:20:05,43] [info] ServiceRegistryActor stopped
[2018-09-14 13:20:05,47] [info] Database closed
[2018-09-14 13:20:05,47] [info] Stream materializer shut down
[2018-09-14 13:20:05,48] [info] WDL HTTP import resolver closed
Workflow caab4283-a3d4-4966-85ba-56d0992c8f00 transitioned to state Failed
(p3cwl) [jeremiah@localhost ~]$

The non-packed CWL is parsed successfully:

(p3cwl) [jeremiah@localhost ~]$ java -jar cromwell/server/target/scala-2.12/cromwell-35-fd560e9-SNAP.jar run gdc-dnaseq-cwl/workflows/bamfastq_align/transform.cwl --inputs gdc-dnaseq-cwl/workflows/bamfastq_align/transform_c9bab596-0b03-4066-b428-dda9cd36fb1c.yml --type CWL
[2018-09-14 13:21:06,87] [info] Running with database db.url = jdbc:hsqldb:mem:ff56e263-4418-491e-afa7-b92aee80129e;shutdown=false;hsqldb.tx=mvcc
[2018-09-14 13:21:16,28] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2018-09-14 13:21:16,30] [info] [RenameWorkflowOptionsInMetadata] 100%
[2018-09-14 13:21:16,42] [info] Running with database db.url = jdbc:hsqldb:mem:66ab1a18-96ee-4819-a2d2-f0e00fbd9b0d;shutdown=false;hsqldb.tx=mvcc
[2018-09-14 13:21:16,95] [info] Slf4jLogger started
[2018-09-14 13:21:17,05] [info] Pre Processing Workflow...
[2018-09-14 13:21:17,25] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/transform.cwl
[2018-09-14 13:21:43,20] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/fastq_clean_pe.cwl
[2018-09-14 13:21:43,34] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/fastq_cleaner_pe.cwl
[2018-09-14 13:21:43,45] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/emit_readgroup_fastq_pe_file.cwl
[2018-09-14 13:21:43,50] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/json_to_sqlite.cwl
[2018-09-14 13:21:43,56] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/fastq_clean_se.cwl
[2018-09-14 13:21:43,65] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/fastq_cleaner_se.cwl
[2018-09-14 13:21:43,70] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/emit_readgroup_fastq_se_file.cwl
[2018-09-14 13:21:43,76] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/merge_sqlite.cwl
[2018-09-14 13:21:43,83] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/readgroups_bam_to_readgroups_fastq_lists.cwl
[2018-09-14 13:21:44,04] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/biobambam2_bamtofastq.cwl
[2018-09-14 13:21:44,22] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/bam_readgroup_to_json.cwl
[2018-09-14 13:21:44,31] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/decider_readgroup_expression.cwl
[2018-09-14 13:21:44,38] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/readgroup_fastq_pe.cwl
[2018-09-14 13:21:44,45] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/emit_json_readgroup_meta.cwl
[2018-09-14 13:21:44,65] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/readgroup_fastq_se.cwl
[2018-09-14 13:21:44,79] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/merge_pe_fastq_records.cwl
[2018-09-14 13:21:44,87] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/merge_se_fastq_records.cwl
[2018-09-14 13:21:44,95] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/bwa_pe.cwl
[2018-09-14 13:21:45,21] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/fastqc.cwl
[2018-09-14 13:21:45,38] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/fastqc_db.cwl
[2018-09-14 13:21:45,46] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/fastqc_basicstatistics_json.cwl
[2018-09-14 13:21:45,54] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/bwa_record_pe.cwl
[2018-09-14 13:21:45,68] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/readgroup_json_db.cwl
[2018-09-14 13:21:45,75] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/bwa_se.cwl
[2018-09-14 13:21:45,98] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/bwa_record_se.cwl
[2018-09-14 13:21:46,12] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_mergesamfiles_aoa.cwl
[2018-09-14 13:21:46,27] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/bam_reheader.cwl
[2018-09-14 13:21:46,35] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/conditional_markduplicates.cwl
[2018-09-14 13:21:46,43] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_markduplicates.cwl
[2018-09-14 13:21:46,53] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_markduplicates_to_sqlite.cwl
[2018-09-14 13:21:46,62] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/conditional_bamindex.cwl
[2018-09-14 13:21:46,71] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_index.cwl
[2018-09-14 13:21:46,80] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/touch.cwl
[2018-09-14 13:21:46,86] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/emit_file_format.cwl
[2018-09-14 13:21:46,90] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/decider_conditional_bams.cwl
[2018-09-14 13:21:46,98] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/gatk4_baserecalibrator.cwl
[2018-09-14 13:21:47,07] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/gatk4_applybqsr.cwl
[2018-09-14 13:21:47,16] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_validatesamfile.cwl
[2018-09-14 13:21:47,29] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_validatesamfile_to_sqlite.cwl
[2018-09-14 13:21:47,37] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/metrics.cwl
[2018-09-14 13:21:48,03] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/amplicon_metrics.cwl
[2018-09-14 13:21:48,15] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collecttargetedpcrmetrics.cwl
[2018-09-14 13:21:48,38] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collecttargetedpcrmetrics_to_sqlite.cwl
[2018-09-14 13:21:48,46] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/exome_metrics.cwl
[2018-09-14 13:21:48,59] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collecthsmetrics.cwl
[2018-09-14 13:21:48,81] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collecthsmetrics_to_sqlite.cwl
[2018-09-14 13:21:48,89] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/gatk4_getpileupsummaries.cwl
[2018-09-14 13:21:48,96] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/gatk4_calculatecontamination.cwl
[2018-09-14 13:21:49,02] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/gatk_calculatecontamination_to_sqlite.cwl
[2018-09-14 13:21:49,10] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectmultiplemetrics.cwl
[2018-09-14 13:21:49,31] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectmultiplemetrics_to_sqlite.cwl
[2018-09-14 13:21:49,48] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectoxogmetrics.cwl
[2018-09-14 13:21:49,59] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectoxogmetrics_to_sqlite.cwl
[2018-09-14 13:21:49,69] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectwgsmetrics.cwl
[2018-09-14 13:21:49,78] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/picard_collectwgsmetrics_to_sqlite.cwl
[2018-09-14 13:21:49,87] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_flagstat.cwl
[2018-09-14 13:21:49,94] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_flagstat_to_sqlite.cwl
[2018-09-14 13:21:50,02] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_idxstats.cwl
[2018-09-14 13:21:50,08] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_idxstats_to_sqlite.cwl
[2018-09-14 13:21:50,17] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_stats.cwl
[2018-09-14 13:21:50,38] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/samtools_stats_to_sqlite.cwl
[2018-09-14 13:21:50,46] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/workflows/bamfastq_align/integrity.cwl
[2018-09-14 13:21:50,64] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/ls_l.cwl
[2018-09-14 13:21:50,71] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/md5sum.cwl
[2018-09-14 13:21:50,81] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/sha256sum.cwl
[2018-09-14 13:21:50,87] [info] Pre-Processing /home/jeremiah/gdc-dnaseq-cwl/tools/integrity_to_sqlite.cwl
[2018-09-14 13:21:51,02] [info] Pre Processing Inputs...
[2018-09-14 13:21:51,27] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-6d01716",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2018-09-14 13:21:51,38] [info] Metadata summary refreshing every 2 seconds.
[2018-09-14 13:21:51,44] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-14 13:21:51,45] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-14 13:21:51,51] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-09-14 13:21:52,32] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-09-14 13:21:52,35] [info] SingleWorkflowRunnerActor: Version 35-fd560e9-SNAP
[2018-09-14 13:21:52,36] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-09-14 13:21:52,42] [info] CWL (Unspecified version) workflow 6f311835-f1fe-4bbd-8fbb-c5543373d039 submitted
[2018-09-14 13:21:52,43] [info] SingleWorkflowRunnerActor: Workflow submitted 6f311835-f1fe-4bbd-8fbb-c5543373d039
[2018-09-14 13:21:52,43] [info] 1 new workflows fetched
[2018-09-14 13:21:53,05] [info] WorkflowManagerActor Starting workflow 6f311835-f1fe-4bbd-8fbb-c5543373d039
[2018-09-14 13:21:53,06] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-09-14 13:21:53,09] [info] WorkflowManagerActor Successfully started WorkflowActor-6f311835-f1fe-4bbd-8fbb-c5543373d039
[2018-09-14 13:21:53,09] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-09-14 13:21:53,10] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-09-14 13:21:53,21] [info] MaterializeWorkflowDescriptorActor [6f311835]: Parsing workflow as CWL v1.0
[2018-09-14 13:21:55,91] [info] MaterializeWorkflowDescriptorActor [6f311835]: Call-to-Backend assignments: bam_ls_l -> Local, bam_readgroup_to_json -> Local, fastqc -> Local, json_to_sqlite -> Local, merge_readgroup_json_db -> Local, fastq_cleaner_se -> Local, picard_collecttargetedpcrmetrics_to_sqlite -> Local, biobambam_bamtofastq -> Local, readgroup_json_db -> Local, picard_collectmultiplemetrics -> Local, bai_integrity_to_db -> Local, gatk_calculatecontamination_to_sqlite -> Local, merge_exome_sqlite -> Local, picard_mergesamfiles -> Local, picard_collecthsmetrics -> Local, merge_sqlite -> Local, fastqc_basicstats_json -> Local, bam_integrity_to_db -> Local, merge_sqlite -> Local, bai_md5sum -> Local, merge_readgroup_json_db -> Local, merge_sqlite_bwa_se -> Local, samtools_index -> Local, picard_validatesamfile_bqsr -> Local, samtools_stats_to_sqlite -> Local, merge_amplicon_sqlite -> Local, bwa_se -> Local, picard_validatesamfile_bqsr_to_sqlite -> Local, fastqc_basicstats_json -> Local, fastqc1 -> Local, merge_readgroup_json_db -> Local, merge_all_sqlite -> Local, readgroup_json_db -> Local, bai_ls_l -> Local, bwa_pe -> Local, picard_collectmultiplemetrics_to_sqlite -> Local, bai_sha256 -> Local, picard_collectoxogmetrics -> Local, bam_sha256 -> Local, bam_readgroup_to_json -> Local, fastqc_db -> Local, fastqc_basicstats_json -> Local, gatk_applybqsr -> Local, merge_sqlite_fastq_clean_se -> Local, picard_markduplicates -> Local, fastq_cleaner_pe -> Local, fastqc2 -> Local, fastqc -> Local, picard_collectwgsmetrics -> Local, merge_sqlite_bwa_pe -> Local, picard_markduplicates_to_sqlite -> Local, fastqc_db -> Local, picard_collecttargetedpcrmetrics -> Local, readgroup_json_db -> Local, bam_readgroup_to_json -> Local, fastqc_db2 -> Local, empty_sqlite -> Local, gatk_calculatecontamination -> Local, fastqc_db1 -> Local, merge_sqlite -> Local, picard_collectoxogmetrics_to_sqlite -> Local, merge_readgroup_json_db -> Local, merge_sqlite -> Local, fastqc -> Local, json_to_sqlite -> Local, readgroup_json_db -> Local, gatk_baserecalibrator -> Local, merge_sqlite -> Local, bam_readgroup_to_json -> Local, picard_collectwgsmetrics_to_sqlite -> Local, bwa_se -> Local, samtools_idxstats -> Local, samtools_idxstats_to_sqlite -> Local, picard_collecthsmetrics_to_sqlite -> Local, merge_sqlite -> Local, fastqc_basicstats_json -> Local, fastqc_db -> Local, samtools_flagstat -> Local, bwa_se -> Local, bam_readgroup_to_json -> Local, samtools_flagstat_to_sqlite -> Local, gatk_getpileupsummaries -> Local, merge_sqlite_fastq_clean_pe -> Local, bam_reheader -> Local, samtools_stats -> Local, bam_md5sum -> Local

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:12 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
cjllanwarnecommented, Jan 8, 2019

Hey all, sorry for my slow response on this one - this is going to be the next item on my list. So, I’ll be having a look at it right after I complete the feature I’m currently looking at. Thanks!

2reactions
geoffjentrycommented, Oct 23, 2018

Hi @jeremiahsavage - I also personally ran into the packed CWL issue at the GA4GH plenary, so can confirm that this is real. I know that this is on the radar of @cjllanwarne but he’s currently looking at another CWL handling issue we ran into at the plenary 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

Common Workflow Language (CWL) Command Line Tool ...
A CommandLineTool is a process characterized by the execution of a standalone, non-interactive program which is invoked on some input, produces output, and...
Read more >
cwl-utils
Packed CWL documents use the $graph construct to contain multiple CWL Process objects (Workflow, CommandLineTool, ExpressionTool, Operation).
Read more >
Common Workflow Language, v1.0
The products of the CWL working group (including this document) are made available ... A workflow is a process characterized by multiple subprocess...
Read more >
Common Workflow Language User Guide: Custom Types
How do I create and import my own custom types into a CWL description? ... between multiple descriptions that all use the same...
Read more >
NetSuite Applications Suite - CSV Import FAQ
Find Plugin Implementations · Generate a SuiteSignOn Token · Load an XML File and Obtain Child Element Values · Parse an XML File...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Slow, expensive metadata endpoint

Next page

Workflows occasionally fail due to timeouts on read_* functions