Proposal: `interpreter` runtime attribute for non-bash tasks

Currently, it is specified that the task command block is executed by a bash interpreter. Non-bash languages are supported either by heredocs or by putting the commands in an external script and calling that script from the command block.

This proposal is to add the optional interpreter attribute to the runtime section. This keyword will be used to specify the interpreter program (name, version, etc) used to execute the command block. This information can be used by the WDL runtime in several ways:

• Execute non-bash scripts without the need for heredocs or external scripts
• Pre-check to verify that the specified interpreter is available in the runtime environment (container or host system) before trying to execute the commands
• Choose a default runtime environment (container, virtualenv, etc) when a container is not specified

Example:

task python_cat_file {
  input {
    File infile
  }

  command <<<
  import sys
  path = "~{infile}"
  print(f"reading from file {path}", file=sys.stderr)
  with open(path, "r") as i:
    print(i.read())
  >>>

  output {
    String content = read_string(stdout())
  }

  runtime {
    interpreter: {
      name: "python",
      version: "^3.6.0",
      variant: "pypy"
    }
  }
}

The interpreter keyword can take one of the following values:

• String with the interpreter name (e.g. “python”) and optional version constraint.

  runtime {
    interpreter: "python ^3.6.0"
  }
  

  • The interpreter name is case-insensitive. It is suggested to use all lowercase.
  • For a programming language with multiple interpreters available, the language name (e.g. “python”) is an alias for the reference/default interpreter variant (e.g. “cpython”).
  • If the version is not specified, the default version is used. The default version may be defined in the WDL specification, or it may be specified by the runtime.
• Array[String] with multiple interpreter variants, in order of preference.

  runtime {
    interpreter: ["cpython ^3.6.0", "PyPy3.6 ^7.3.0"]
  }
  

• Object with the following keys:
  • name: The interpreter name. Required.
  • version: An optional version constraint.
  • variant: An optional interpreter variant.
  • args: An optional Array[String] with arguments to pass to the interpreter when executing the command. The runtime may specify interpreter arguments that are always used and/or not allowed - any arguments specified by the args keyword are merged with the required arguments, and an error is raised if any of the user-specified arguments are disallowed.

  runtime {
    interpreter: {
      name: "bash",
      version: "^5.0",
      args: ["--posix"]
    }
  }
  

• Array[Object] with multiple Object-style interpreter variants.

Additional rules:

• If no interpreter keyword is specified, the default value is "bash".
• If the interpreter keyword is specified, the host system must provided an interpreter that is compatible with one of the user’s specifications, or raise an error if no compatible interpreters are available.
• If the user specifies the container runtime attribute, then one of the specified interpreters must be available in that container.
  • If container specifies an array of containers, then the runtime may (but is not required to) use the availability of one of the specified interpreters within the container as a criteria in selecting which of the containers to use.
• If the user does not specify the container runtime attribute then the runtime may try either or both of the following, in any order. In either case, no other dependencies are guaranteed to exist aside from the interpreter.
  • Execute the commands on the host system by searching for a compatible interpreter
  • Execute the commands in a container of the runtime’s choice that provides one of the specified interpreters

[rhpvorderman]

Could you explain why the heredoc syntax would not work in this case?

I am not in favour of adding extra syntax and complexity to WDL if there is not a clear benefit.

On all these issues I tend to say: make sure you have some sort of container, so you have full control of the interpreter. And either:

• Run the script in heredoc
• If that is unwieldly, you probably created a tool. So then make it a proper tool with tests and such and then create a container so you can use one command instead of a script.

If the “only bash” policy forces users to make proper tools instead of making WDLs full of inline lisp scripts, I’d say that is a win for WDL as a language. I come from a snakemake background, and that was truly awful. Users were granted so much freedom that no two snakemake pipelines looked alike. Hence it was impossible to learn. Everyone rolled their own custom solution to the same problems. So I am very conservative when it comes to adding stuff to WDL that even remotely hints at allowing the same freedom.

I get @jdidion 's argument that the execution engine can solve which interpreter is needed. But I think that is already handled by containers. If you run your job outside of a container that is bad practice, and there should not be any syntax to make it seem like running outside a container is a viable option.

[pshapiro4broad]

I agree with @rhpvorderman 's last comment. In the WDL I’ve worked on the current behavior hasn’t been a big enough obstacle to require a change to the WDL language.

I kind of like @patmagee 's suggestion to support a shebang line, but one downside I can see is that it requires the WDL runtime to read the first line of the command block and interpret it, possibly using string interpolation. Placing it outside the block would make it more clearer that the WDL runtime will handle it specially and that it’s not really part of the command script.