Keep going with independent tasks

When a WDL task fails (run via AGC), I’ve observed that all other tasks are killed. I’d like miniwdl to execute as many tasks as possible until there are no more tasks that are independent of the failure(s). See snakemake’s --keep-going option and Nextflow’s errorStrategy. This would allow me to complete as many tasks as possible, so that the next time I re-run the workflow while caching tasks, I’ll be closer both to completion as well as having the task that failed start sooner (to see if it works this time).

[nh13]

Thanks @mlin this is super helpful and thank you for your patience. I’m really enjoying using miniwdl so hopefully these questions are not seen as a criticism but a desire to understand, and contribute back in some small part.

[mlin]

There’s a config option [scheduler] fail_fast = false / env MINIWDL__SCHEDULER__FAIL_FAST=false that should do this. I don’t think we have a test case for it with AWS specifically, but that scheduler logic is “above” the container backend. Let me know if it doesn’t work.