assumptions around modifying input files, or making new files in their directories

It is suggested the the OpenWDL specification clarify the write-status of input files, and the directories they are in.

I suggest that in a future version of the OpenWDL spec, it is declared that all inputs files must be make read-only to have consistent behavior.

This also helps with converting to CWL, as it has the same restriction, unless InitialWorkDirRequirement is used to mark some inputs as writable: true

For miniwdl, they have an IO-expensive workaround https://miniwdl.readthedocs.io/en/latest/runner_advanced.html#read-only-input-files

[rhpvorderman]

MUST is the only option in my opinion. Mostly because WDL strives to be readable. Having to keep in mind that a task manipulates the inputs adds very significant cognitive overhead. Even seemingly simple workflows might be not as simple as they seem, so each individual task needs to be inspected to ensure that indeed, only the outputs are affected by the tasks, not the inputs.

I think that is a very undesirable state for WDL to be in. Reading a workflow from the top-level should be enough to infer what is happening.

[rhpvorderman]

Regarding “making new files in their directories” – IIRC (correction welcomed) Cromwell drops input files into the task working directory, so I think practically this has to be allowed.

Cromwell creates a separate inputs directory that is in the parent of the execution (working) directory. All files are solved to absolute paths in cromwell.

As for the samtools index case: BioWDL makes a hardlink this works in Cromwell because inputs and execution are always on the same filesystem. It is an extremely ugly hack, and precisely for that reason we don’t use the samtools index task. Usually when handling BAM files, the utility that does so has an index command. So we just perform the indexing directly in the command that produces a new BAM file. That solves a lot of hassle. It is really only a problem for tools that have no flag for specifying the index if it is not in the same dir. Maybe we should fix this issue there?

Anyway, sorry for the digression. I am in favor of enforcing read-only inputs. In terms of reproducibility, if I run task A, and then run task A again on the same input, that should have the same result. That can not be guaranteed if task A changes the input.