Output directory structure

Hi,

After the testathlon, I thought it would be nice to add miniwdl to the pytest-workflow functional tests and documentation.

In the process of doing so I found that miniwdl has a slightly surprising output directory structure. The workflow I run for functional testing can be found in this wdl, the inputs are in this json, and the pytest-workflow tests are in this yaml.

When I run these tests with cromwell’s final_workflow_outputs_dir functionality and enable the relative_output_paths, all the files end up where I expect them.

In miniwdl there is a consistent output directory structure which is suitable for testing if the -d flag is used. However, the placing of the files is a bit unintuitive.

In cromwell the files end up like this

    - path: test-output/rand/0.bin
    - path: test-output/rand/1.bin
    - path: test-output/rand/2.bin
    - path: test-output/rand/3.bin
etc.

and in miniwdl like this

    - path: test-output/output_links/rand_files/0/0.bin
    - path: test-output/output_links/rand_files/1/1.bin
    - path: test-output/output_links/rand_files/2/2.bin
    - path: test-output/output_links/rand_files/3/3.bin
etc.

This makes perfect sense from a file collision perspective. But on the other hand I do not get the directory structure I intended. The subfolders I defined are gone, there are sudden numbered directories, there is a dir called <task_name>_files and there is an output_links directory. Also, all the call folders are in the output.

As a user I am only interested in the results, and not in the internal structure of miniwdl. Would it be desirable to have a similar ‘relative output’ function for miniwdl so I can get a folder with just the results, like I can in cromwell?

If you are okay with this I can start working on a PR.

Best regards, Ruben

[rhpvorderman]

A miniwdl example will be added to pytest-workflow (https://github.com/LUMC/pytest-workflow/pull/94) and should be available in the next release. Thanks again!

[rhpvorderman]

Thanks. That is clear! The directory structure as it is currently is testable, so I will document that in the pytest-workflow miniwdl example documentation.