SGE default config redirects stdout,stderr to the same file twice

https://github.com/broadinstitute/cromwell/blob/215cca97e6efafa7406fd89b38e21a39ce2d0d4e/cromwell.examples.conf#L440

It seems like the default configuration sends the stdout and stderr to the same filenames twice. In the config above, qsub is called with -o EG: e1bd0801-52c1-445c-8513-72e9872a0c7d/call-thing/execution/stdout

In the script that will be executed on the node, there is some shenanigans:

(
cd .../e1bd0801-52c1-445c-8513-72e9872a0c7d/call-thing/execution


bla bla bla my command invocation
)  > >(tee '.../e1bd0801-52c1-445c-8513-72e9872a0c7d/call-thing/execution/stdout') 2> >(tee '.../e1bd0801-52c1-445c-8513-72e9872a0c7d/call-thing/execution/stderr' >&2)

Those ‘tees’ are generated by cromwell somewhere, I do not know if the config has control of that. I don’t know the details of what will happen, but I do not think it will be healthy. I smell race conditions. I came across this trying to debug missing log data when SGE aborts a job.

[mcovarr]

Thanks for reporting this, you’re right that those files should not have the same name. The files within the script are meant to capture the stdout and stderr of the user command only and are required for both WDL and CWL support. The files on the qsub command line are meant to capture all stdout and stderr including those log messages. The structure of the script has changed somewhat with Cromwell 32 but I suspect this problem still exists so we’ll look at making a fix.

[EvanTheB]

thanks guys