Exception in task_file_mover when ingressing files from other batch tasks

I’ve set up a job which contains several fetch tasks, and a single processing task that depends on the fetch tasks. For convenience, I tried using the Azure Batch input_data type in the processing task to get all the data from the preceding fetch tasks, but I’m running into this exception with task_file_mover.

Traceback (most recent call last):
  File "task_file_mover.py", line 148, in <module>
    main()
  File "task_file_mover.py", line 123, in main
    batch_client = _create_credentials()
  File "task_file_mover.py", line 60, in _create_credentials
    ba, url, bakey = os.environ['SHIPYARD_BATCH_ENV'].split(';')
ValueError: not enough values to unpack (expected 3, got 2)

I’m using KeyVault for supplying the batch credentials, like:

{
    "credentials": {
        "batch": {
            "account": "myaccount",
            "account_key_keyvault_secret_id": "https://myvault.vault.azure.net/secrets/batchkey",
            "account_service_url": "https://myaccount.westus.batch.azure.com"
        }
    }
}

[sebastko]

I have validated that your fix in the “develop” branch has solved my problem - thank you very much! I will also remember to use “depends_on” - thank you for pointing out! (perhaps you can consider updating documentation at http://batch-shipyard.readthedocs.io/en/latest/14-batch-shipyard-configuration-jobs/ where input_data -> azure_batch is mentioned?)

[alfpark]

For the third point, you will want to enforce task dependencies. You are getting “lucky” in this case as the download_data task is getting scheduled ahead of the other task that requires those files and it is blocked for scheduling for reasons such as insufficient available slots on compute nodes.