Launching workflows using WES, FileNotFoundError.

I’m currently attempting to setup a local WES server and launch workflows with toil by using the docker-compose section of this documentation page: https://toil.readthedocs.io/en/latest/running/server/wes.html. This appears to function correctly at first, as the containers are running and I can execute basic WES requests against the server (Such as getting the server info and listing the run status of a workflow).

The issue appears when launching a workflow. Launching workflows returns a run ID (as expected), but the workflow hangs in the QUEUED state forever. Taking a look at the server logs, it seems the celery-worker does not have the proper workflow directory files:

celery-worker  |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
celery-worker  | [2022-01-31 17:45:24,355: ERROR/ForkPoolWorker-2] Task run_wes[8c9f1f3368d744ffb54a6f73eeb7cefd] raised unexpected: FileNotFoundError(2, 'No such file or directory')
celery-worker  | Traceback (most recent call last):
celery-worker  |   File "/app/src/toil/src/toil/server/wes/tasks.py", line 283, in run_wes
celery-worker  |     runner.run()
celery-worker  |   File "/app/src/toil/src/toil/server/wes/tasks.py", line 192, in run
celery-worker  |     self.set_state("INITIALIZING")
celery-worker  |   File "/app/src/toil/src/toil/server/wes/tasks.py", line 64, in set_state
celery-worker  |     safe_write_file(os.path.join(self.work_dir, "state"), state)
celery-worker  |   File "/app/src/toil/src/toil/lib/retry.py", line 256, in call
celery-worker  |     return func(*args, **kwargs)
celery-worker  |   File "/app/src/toil/src/toil/server/utils.py", line 103, in safe_write_file
celery-worker  |     file_obj = open(file, "r+")
celery-worker  | FileNotFoundError: [Errno 2] No such file or directory: '/app/workflows/8c9f1f3368d744ffb54a6f73eeb7cefd/state'
celery-worker  | 
celery-worker  | During handling of the above exception, another exception occurred:
celery-worker  | 
celery-worker  | Traceback (most recent call last):
celery-worker  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
celery-worker  |     R = retval = fun(*args, **kwargs)
celery-worker  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
celery-worker  |     return self.run(*args, **kwargs)
celery-worker  |   File "/app/src/toil/src/toil/server/wes/tasks.py", line 294, in run_wes
celery-worker  |     runner.set_state("EXECUTOR_ERROR")
celery-worker  |   File "/app/src/toil/src/toil/server/wes/tasks.py", line 64, in set_state
celery-worker  |     safe_write_file(os.path.join(self.work_dir, "state"), state)
celery-worker  |   File "/app/src/toil/src/toil/lib/retry.py", line 256, in call
celery-worker  |     return func(*args, **kwargs)
celery-worker  |   File "/app/src/toil/src/toil/server/utils.py", line 103, in safe_write_file
celery-worker  |     file_obj = open(file, "r+")

Taking a look inside the containers, I can see the following:

1. The wes-server container has the appropriate workflow files at /app/workflows/8c9f1f3368d744ffb54a6f73eeb7cefd/state
2. The celery-worker container does not have the /app/workflows directory (and thus, none of the child files)

---

To reproduce:

1. Go to https://toil.readthedocs.io/en/latest/running/server/wes.html
2. Create the Dockerfile and docker-compose.yml files as specified in https://toil.readthedocs.io/en/latest/running/server/wes.html#running-the-server-with-docker-compose
3. Run docker compose up (Not using --detach to easily see logs)
4. Create example.cwl using the example provided in https://toil.readthedocs.io/en/latest/running/server/wes.html#running-the-server-with-docker-compose
5. Execute the cURL command:

curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs' \
    --form 'workflow_url="example.cwl"' \
    --form 'workflow_type="cwl"' \
    --form 'workflow_type_version="v1.0"' \
    --form 'workflow_params="{\"message\": \"Hello world!\"}"' \
    --form 'workflow_attachment=@"./toil_test_files/example.cwl"'

1. After a few moments, the celery-worker will receive the request and then error out due to the missing file.

---

I’m wondering if the WES documentation page is slightly out-of-date, and if the docker-compose.yml or Dockerfile examples should be reconfigured? Not sure, just brainstorming. Thanks!

┆Issue is synchronized with this Jira Story ┆friendlyId: TOIL-1138

[Richard-Hansen]

Thanks @w-gao , I was able to successfully run the docker compose file. I did run into some other issues afterwards, so added some comments to the linked PR: https://github.com/DataBiosphere/toil/pull/4027#pullrequestreview-872057152

[Richard-Hansen]

Thanks for the help, my issue has been resolved in: https://github.com/DataBiosphere/toil/pull/4027 See comment: https://github.com/DataBiosphere/toil/pull/4027#discussion_r800881954