Generate WDL from a parsed tree

Hi Mike,

One external feature we’re looking to implement in the upcoming DSP hackathon is collection of runtime monitoring data from individual tasks in WDL. The metrics would be reported from each individual task call on PAPIv2, using a custom monitoring_image that records time-series of cpu/memory/disk utilization to BigQuery. There’s a (somewhat outdated) proposal for this in https://github.com/broadinstitute/cromwell-task-monitor-bbq (I believe in our current thinking, it could be implemented w/o modifying Cromwell).

This data will then be used to build regression models that predict the actual amount of resources required for a given task, given its inputs. To collect training data however, we may need to do a “sweep”/“ladder” on the runtime parameters. If we could modify the runtime parameters inside the WDL without having to supply them as task inputs (which would require involvement of the workflow developer), this would provide a fully automated way to optimize a given workflow, without having to supply anything other than a collection of actual workflow inputs.

So in summary, it would be helpful to replace runtime attributes for a given task, and render the workflow tree back into a .wdl file, without any other modifications to the workflow itself. Do you think this feature could be accomplished by MiniWDL? Or do you have any alternatives we could use to achieve that?

EDIT: And as a bonus, we could also make this part of LSP feedback to the user. So if their workflow ran on PAPIv2, we could provide “auto-suggestions” on the calculated amount of resources for a given task (like suggest replacing memory: '2G' with memory: ceil(2.34 * size(in_bam, 'G')).

Thanks a lot!

CC @rexwangcc @mohawkTrail

[mlin]

Just-released v0.7.3 finally includes source code comments in the miniwdl AST, so we’re one step closer here although wdlTools also looks to be coming along nicely.

[jdidion]

I have a prototype here: https://github.com/jdidion/wdlkit

After I finished this I found out DNAnexus already has a toolkit that’s being developed in Scala (https://github.com/dnanexus-rnd/wdlTools). Therefore, I plan to port what I have to that project and discontinue the wdlkit project. @mlin if you’re happy with the syntax formatter in wdlkit I’m happy to donate it to miniwdl. Alternatively, if anyone wants to take ownership of wdlkit and keep developing it as a separate package, I’m happy to hand over the keys.