Issue when sourcing `shipyard.envlist`

Problem Description

When sourcing the .shipyard.envlist, I got an the following errors :

/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 4: --name: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^SHLVL=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^HOSTNAME=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^PATH=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^PWD=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^HOME=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 11: ^USER=: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 16: -c: command not found
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 20: source /mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist && run_freesurfer.sh : No such file or directory
/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd/.shipyard.envlist: line 29: -c: command not found

Batch Shipyard Version

We recently upgraded from batch-shipyard 3.5.0 to batch-shipyard 3.8.1

Steps to Reproduce

Submit a job with batch-shipyard 3.8.1 with a task sourcing the shipyard.envlist

Expected Results

No errorwhen sourcing the shipyard.envlist

Actual Results

Error when sourcing the shipyard.envlist

Redacted Configuration

batch_shipyard:
  storage_account_settings: shipyardstorage

global_resources:
  additional_registries:
    docker:
    - micliqs.azurecr.io

Additional Logs

Content of the shipard.envlist on the node

AZ_BATCH_TASK_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer
SHIPYARD_ENV_FILE=.shipyard.envlist
NB_CORE=2
SHIPYARD_RUNTIME_CMD_OPTS=--rm --name freesurfer_CADASIL_1001_Baseline_3DT1-Freesurfer --entrypoint /bin/bash -w $AZ_BATCH_TASK_WORKING_DIR -v $AZ_BATCH_NODE_ROOT_DIR:$AZ_BATCH_NODE_ROOT_DIR --env-file .shipyard.envlist
AZ_BATCH_NODE_STARTUP_DIR=/mnt/batch/tasks/startup
DOCKER_LOGIN_PASSWORD==redacted
AZ_BATCH_CERTIFICATES_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/certs
AZ_BATCH_ACCOUNT_URL=https://****.westeurope.batch.azure.com/
AZ_BATCH_TASK_WORKING_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/Freesurfer/wd
AZ_BATCH_NODE_SHARED_DIR=/mnt/batch/tasks/shared
SHIPYARD_ENV_EXCLUDE=^_=|^SHLVL=|^HOSTNAME=|^PATH=|^PWD=|^HOME=|^USER=
AZ_BATCH_JOB_PREP_WORKING_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/jobpreparation/wd
DOCKER_LOGIN_SERVER=****.azurecr.io
AZ_BATCH_TASK_USER=_azbatch
AZ_BATCH_NODE_ROOT_DIR=/mnt/batch/tasks
SHIPYARD_SYSTEM_PROLOGUE_CMD=/bin/bash -c 'set -e; set -o pipefail; pushd /mnt/batch/tasks/startup/wd; ./registry_login.sh; popd; set -f; $AZ_BATCH_NODE_STARTUP_DIR/wd/shipyard_blobxfer.sh "1.9.1,i,False,redacted,$AZ_BATCH_TASK_WORKING_DIR,--include "*.dcm" "; set +f; wait'
AZ_BATCH_JOB_ID=freesurfer_CADASIL_1001_Baseline_3DT1
AZ_BATCH_NODE_IS_DEDICATED=true
INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/cadasil/3dd5e7dd-29d4-4699-97d2-c73b2bcdf434
SHIPYARD_USER_CMD=-c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist && run_freesurfer.sh "
AZ_BATCH_JOB_PREP_DIR=/mnt/batch/tasks/workitems/freesurfer_CADASIL_1001_Baseline_3DT1/job-1/jobpreparation
SUBJECT_NAME=3dd5e7dd-29d4-4699-97d2-c73b2bcdf434
SHIPYARD_CONTAINER_IMAGE_NAME=****.azurecr.io/freesurfer:1.4.1
AZ_BATCH_NODE_MOUNTS_DIR=/mnt/batch/tasks/fsmounts
AZ_BATCH_NODE_ID=tvm-605534940_8-20190916t100009z
SHIPYARD_RUNTIME=docker
AZ_BATCH_POOL_ID=iqsfreesurfer
AZ_BATCH_TASK_ID=Freesurfer
SHIPYARD_SYSTEM_EPILOGUE_CMD=/bin/bash -c 'set -e; set -o pipefail; set -f; $AZ_BATCH_NODE_STARTUP_DIR/wd/shipyard_blobxfer.sh "1.9.1,e,False,redacted,$AZ_BATCH_TASK_WORKING_DIR/3dd5e7dd-29d4-4699-97d2-c73b2bcdf434,--include "*.tar.gz" ,tasksuccess"; set +f; wait'
DOCKER_LOGIN_USERNAME=***
AZ_BATCH_ACCOUNT_NAME=***
SHIPYARD_RUNTIME_CMD=run
AZ_BATCH_TASK_USER_IDENTITY=PoolAdmin
OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR

Additonal Comments

Thank you for your help

[f-roche]

Thanks for your help, we decided to change it like this at the end:

• previous

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
      INPUT_DICOM_DIR: $AZ_BATCH_TASK_WORKING_DIR/foo/bar
      OUTPUT_DIR: $AZ_BATCH_TASK_WORKING_DIR
    entrypoint: /bin/bash
    command: -c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist &&
                 run.sh {options}"

• now

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
    entrypoint: /bin/bash
    command: -c "export INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/foo/bar &&
                 export OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR &&
                 run.sh {options}"

We are using the shell to do the expansion before assigning environment variables.

[0zzzhhui]

Thanks for your help, we decided to change it like this at the end:

• previous

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
      INPUT_DICOM_DIR: $AZ_BATCH_TASK_WORKING_DIR/foo/bar
      OUTPUT_DIR: $AZ_BATCH_TASK_WORKING_DIR
    entrypoint: /bin/bash
    command: -c "source $AZ_BATCH_TASK_WORKING_DIR/.shipyard.envlist &&
                 run.sh {options}"

• now

- id: task1
    docker_image: ***********/**:{version}
    environment_variables:
      SUBJECT_NAME: foobar
    entrypoint: /bin/bash
    command: -c "export INPUT_DICOM_DIR=$AZ_BATCH_TASK_WORKING_DIR/foo/bar &&
                 export OUTPUT_DIR=$AZ_BATCH_TASK_WORKING_DIR &&
                 run.sh {options}"

We are using the shell to do the expansion before assigning environment variables.

I am a newer for develop batch shipyard, can you tell me where the {version} is defined? I need to define a docker image variable in my job config yaml file.