Turning WDL files into executables using Cromexe

Hi everyone,

I got an idea (read “stole it from another tool”) that WDL files can easily be made executable using shebang lines.

Instead of calling:

java -jar ${CROMWELL_JAR} run example/hello.wdl -i example/hello.input.json

By using a #!/usr/bin/env cromwexe shebang line in our WDL file, we can call simply:

example/hello.wdl -i example/hello.input.json

See my working prototype: https://github.com/prihoda/cromwexe

The only thing cromwexe is doing at the moment is passing all args to the regular java command:

cmd = ["java", "-jar", cromwell_jar, "run"] + sys.argv[1:]

It starts to get interesting once you realize that if we can parse the workflow script inside cromwexe, we can automatically generate a command-line interface that parses the args directly from the command line (and even provide a help screen with descriptions of all input args):

example/hello.wdl -i example/hello.defaults.json --sayHello.name World

We can also parse the job metadata output of cromwell to determine the location of our output files and move them over from the execution folder to the user’s working folder, if the user provides the output arg as well:

example/hello.wdl -i example/hello.defaults.json --sayHello.name World --output.greeting greeting.txt

Edit: I see that this can be done by the use_relative_output_paths or final_workflow_outputs_dir args. Then I guess the added functionality is that output args could be used to define each output file path separately.

I have two questions 😃

1. What do you think? I am excited to implement the full solution, it should not take more than a few hours.
2. Can I use this parser? https://github.com/TMiguelT/WdlParserPackaging

[mlin]

@prihoda

miniwdl has a little CLI wrapper to make it nicer to launch cromwell locally. It doesn’t do the the shebang script which is a neat idea, however, it does implement versions of (i) parsing the task/workflow inputs to expose them as command-line arguments, and (ii) parsing the outputs to organize them more nicely after they come out. Here is a link to the CLI entrypoint for this where you can see how all this happens. I’d be happy to work with you on merging & fleshing out the ideas.

[dinvlad]

Additionally, @rexwangcc has been working on https://cromwell-tools.readthedocs.io, which at least partially seems to implement what you proposed. We’re also planning to move other functionality, like parsing of metadata.json for failures, into it over time, to be used together with miniWDL for WDL debugging support.