Unable to substitute plain text password with environment variable in node.conf

Corda: Open Source 4.3 Operating System: Ubuntu 18.04.3 LTS

I’m trying to substitute my plain text password with an environment variable with the following steps:

1. Create a secret on AWS Secrets Manager with one key/value: keyStorePassword
2. Create a file called corda_node_config.sh under /etc/profile.d to permanently persist the environment variable. Inside that file I have:

export keyStorePassword=$(aws secretsmanager get-secret-value --region my-region 
--secret-id my-secret | jq --raw-output '.SecretString' | jq -r '.keyStorePassword')

1. I exit my VM and log back. I can see the value when I run echo $keyStorePassword
2. Inside my node.conf I replace the plain text with the environment value: keyStorePassword = ${keyStorePassword}
3. Now when I start the node, I get:

[ERROR] 21:21:14+0000 [main] internal.NodeStartupLogging. - Exception during node startup: 
Keystore was tampered with, or password was incorrect [errorCode=ikfywq, moreInformationAt
=https://errors.corda.net/OS/4.3/ikfywq]

1. I am 100% sure the environment variable is correct; when I run this, I don’t get any errors:

keytool -list -keystore nodekeystore.jks -storepass $keyStorePassword
keytool -list -keystore sslkeystore.jks -storepass $keyStorePassword

1. If I remove the environment variable and put back the plain text value, my node starts; so the keystore hasn’t been tampered with.

Full log:

[ERROR] 2020-02-17T21:21:14,300Z [main] internal.NodeStartupLogging. - Exception during node startup: Keystore was tampered with, or password was incorrect [errorCode=ikfywq, moreInformationAt=https://errors.corda.net/OS/4.3/ikfywq]
java.io.IOException: Keystore was tampered with, or password was incorrect
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:782) ~[?:1.8.0_241]
	at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[?:1.8.0_241]
	at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_241]
	at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[?:1.8.0_241]
	at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_241]
	at net.corda.nodeapi.internal.crypto.KeyStoreUtilities.loadOrCreateKeyStore(KeyStoreUtilities.kt:27) ~[corda-node-api-4.3.jar:?]
	at net.corda.nodeapi.internal.crypto.X509KeyStore$Companion.fromFile(X509KeyStore.kt:32) ~[corda-node-api-4.3.jar:?]
	at net.corda.nodeapi.internal.config.CertificateStore$Companion.fromFile(CertificateStore.kt:20) ~[corda-node-api-4.3.jar:?]
	at net.corda.nodeapi.internal.config.FileBasedCertificateStoreSupplier.get(CertificateStoreSupplier.kt:22) ~[corda-node-api-4.3.jar:?]
	at net.corda.nodeapi.internal.cryptoservice.bouncycastle.BCCryptoService.<init>(BCCryptoService.kt:33) ~[corda-node-api-4.3.jar:?]
	at net.corda.nodeapi.internal.cryptoservice.CryptoServiceFactory$Companion.makeCryptoService(CryptoServiceFactory.kt:18) ~[corda-node-api-4.3.jar:?]
	at net.corda.node.internal.AbstractNode.<init>(AbstractNode.kt:183) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.AbstractNode.<init>(AbstractNode.kt:128) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.Node.<init>(Node.kt:102) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.Node.<init>(Node.kt:101) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartup.createNode(NodeStartup.kt:195) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartup$initialiseAndRun$5.invoke(NodeStartup.kt:187) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartup$initialiseAndRun$5.invoke(NodeStartup.kt:138) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartupLogging$DefaultImpls.attempt(NodeStartup.kt:529) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartup.attempt(NodeStartup.kt:138) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartup.initialiseAndRun(NodeStartup.kt:186) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartupCli.runProgram(NodeStartup.kt:129) ~[corda-node-4.3.jar:?]
	at net.corda.cliutils.CordaCliWrapper.call(CordaCliWrapper.kt:190) ~[corda-tools-cliutils-4.3.jar:?]
	at net.corda.node.internal.NodeStartupCli.call(NodeStartup.kt:84) ~[corda-node-4.3.jar:?]
	at net.corda.node.internal.NodeStartupCli.call(NodeStartup.kt:65) ~[corda-node-4.3.jar:?]
	at picocli.CommandLine.execute(CommandLine.java:1173) ~[picocli-3.9.6.jar:3.9.6]
	at picocli.CommandLine.access$800(CommandLine.java:141) ~[picocli-3.9.6.jar:3.9.6]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1367) ~[picocli-3.9.6.jar:3.9.6]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:1335) ~[picocli-3.9.6.jar:3.9.6]
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) ~[picocli-3.9.6.jar:3.9.6]
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) ~[picocli-3.9.6.jar:3.9.6]
	at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73) ~[corda-tools-cliutils-4.3.jar:?]
	at net.corda.node.Corda.main(Corda.kt:13) ~[corda-node-4.3.jar:?]
Caused by: java.security.UnrecoverableKeyException: Password verification failed
	at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ~[?:1.8.0_241]
	... 32 more

[adelrustum]

Thanx Matthew for your reply,

Honestly I moved on from this issue and used a different approach, you can read about my setup here: https://blog.b9lab.com/enabling-corda-security-with-nodes-configuration-file-412ce6a4371c

If the method that you mentioned works, can you please update the documentation here: https://docs.corda.net/node-administration.html#id2

[mnesbit]

We originally imported all environment variables into the node.conf, but that was problematic and risked exposing secrets in log files. As a result we require that environment variables are prefixed with corda_ or CORDA_ and we strip this off when substituting, so variables in the node.conf are listed without the prefix e.g. export CORDA_MY_PASSWORD=password and in node.conf keyStorePassword = ${MY_PASSWORD}

Can you try this and see if that works properly