Error on mdapi:deploy asking for location of "server.key"

Summary

I have set up a Jenkins pipeline to automatically deploy to orgs using sfdx cli. The errors which I have been having recently (they just started to happen since 21st May) is

ERROR running force:mdapi:deploy:  ENOENT: no such file or directory, open '/apps/tempjenkins/workspace/y_Package_Multi_Pipeline_develop@tmp/secretFiles/d446ee9f-22e7-4b4c-8f39-2117c88b29ac/server.key'

The authentication to the target org, which will always be performed as the first step for every pipeline job, was fine according to the log.

2019-06-28T03:01:57.428Z] + sfdx force:auth:jwt:grant --clientid **** --jwtkeyfile **** --username xxxxxxxxxxx --setalias sit --instanceurl https://test.salesforce.com

[2019-06-28T03:01:57.797Z] Successfully authorized xxxxxxxx with org ID 00D0p0000008agxxxx

JWT keyfile is supplied using withCredentials in this context.

The only workaround we could do is to uninstall sfdx and reinstall it and everything will work fine for a period of time before this error started coming up again.

Steps To Reproduce:

1. Try jenkins with the following snippet of code:

withCredentials([file(credentialsId: 'SIT-SECRET-FILE', variable: 'jwt_key_file')]) {
                    sh 'sfdx force:auth:jwt:grant --clientid ${sitConsumerKey} --jwtkeyfile ${jwt_key_file} --username xxxxxxxx --setalias sit --instanceurl https://test.salesforce.com'
                    sh "sfdx essentials:filter-metadatas -i ./src -o ./package -p ./src/package.xml"
                    sh(returnStdout: false, script: '''
                        if [ -d ./package/classes ] 
                        then 
                            sfdx force:mdapi:deploy -d ./package -u sit -l RunSpecifiedTests -w -1 -g -r \$(sh testclasses.sh ./package/classes)
                        else 
                            sfdx force:mdapi:deploy -d ./package -u sit -l NoTestRun -w -1 -g 
                        fi
                    '''.stripIndent())
                }

Remark: I only pasted a snippet of the problematic code here not the full code, because the full code would contain references to libraries and steps which are unique to my Jenkins environment.

1. Run the Jenkins job

Expected result

The mdapi:deploy command should work properly without errors.

Actual result

ERROR running force:mdapi:deploy:  ENOENT: no such file or directory, open '/apps/tempjenkins/workspace/y_Package_Multi_Pipeline_develop@tmp/secretFiles/d446ee9f-22e7-4b4c-8f39-2117c88b29ac/server.key'

The command failed with the error above without even reaching to Salesforce org.

Additional information

Feel free to attach a screenshot.

SFDX CLI Version sfdx-cli/7.13.0-27dbcb37d3 linux-x64 node-v10.15.3

[cumulus-robert]

My issue has been resolved with the feedback of ensuring scripts are setting home environment local to the job vs. global home environment.

Wrapping my ‘With credentials’ groovy script with:

withEnv(["HOME=${env.WORKSPACE}"]) {
    withCredentials(
    ....
    )
}

For Jenkins jobs outside of pipeline, adding home environment specification (export HOME=$WORKSPACE) to the script fixed my outside Jenkins jobs:

ssh -T [e-mail]
/opt/bitnami/git/bin/git config --global user.email "[e-mail]"
/opt/bitnami/git/bin/git config --global user.name "Jenkins"
export HOME=$WORKSPACE
sfdx force:auth:jwt:grant --clientid $CONNECTED_APP_CONSUMER_KEY --username $username --jwtkeyfile //$jwt_key_file --setdefaultdevhubusername --instanceurl $sandbox -a $username
#make sure project file points to sandbox
cat sfdx-project.json|jq '.sfdcLoginUrl = "https://test.salesforce.com"' > sfdx-project.json1
mv sfdx-project.json1 sfdx-project.json
#retrieve source code from sandbox/prod
sfdx force:source:retrieve -p $WORKSPACE/force-app/main/default/ -u $username
#commit any changes found to version control
/opt/bitnami/git/bin/git status
/opt/bitnami/git/bin/git add .
/opt/bitnami/git/bin/git commit -a -m "Pulled down changes from full sandbox." || true

Thank you to @clairebianchi and team for taking the time to review and ultimately resolve my issue.

[shetzel]

The fix for this issue will be included in the patch release this Thursday, 1/9/2020. Apologies for the delay in getting the fix out and for all the CI headaches it’s caused!