NMS pulls TLS certs from wrong location on Vault

Describe the bug I try to deploy the example corda network to Amazon EKS (1.19). All goes well up to the point that the NMS needs to boot up. I have inspected the NMS pod logs and it cannot pull the tlscerts from the instructed location:

│ init-certificates-tls VAULT TOKEN IS : s.L6mxgxOy74b6sseMWPI6urUF                                                                                                                                                │
│ init-certificates-tls { "errors": [] }                                                                                                                                                                           │
│ init-certificates-tls ERROR: unable to retrieve /data/tlscerts: {                                                                                                                                                │
│ init-certificates-tls   "errors": []                                                                                                                                                                             │

It seems natural that this fails, because when the TLS certs for the NMS are generated, they are placed in the networkmap/certs dir on the Vault instead of networkmap/tlscerts.

TASK [create/certificates/nms : Putting certs and credential to vault for
networkmap] ******************************************************************************************************************************
task path:
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/roles/create/certificates/nms/tasks/main.yaml:179
changed: [localhost] => {"changed": true, "cmd": "vault kv put networkmap/credentials/mongodb
mongodbPassword=\"newdbnm\"\nvault kv put networkmap/credentials/userpassword sa=\"admin\"\nvault kv
put networkmap/certs networkmap.jks=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/cordanetworkmap/keys.jks |
base64)\" rootcakey=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/cordarootca/keys.jks |
base64)\" cacerts=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/cordarootca/cordarootca.pem |
base64)\" keystore=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/cordarootca/cordarootca.key |
base64)\" mongodb-networkmap.pem=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/mongodbca/mongodb-networkmap.pem |
base64)\" mongoCA.crt=\"$(cat
/Users/andrevanderheijden/Documents/mobifi/bevel/platforms/r3-corda/configuration/build/corda/mongorootca/mongoCA.crt |
base64)\"\n", "delta": "0:00:01.321277", "end": "2022-01-27 08:45:48.025138", "msg": "", "rc": 0,
"start": "2022-01-27 08:45:46.703861", "stderr": "", "stderr_lines": [], "stdout": "Key Value\n---
-----\ncreated_time 2022-01-27T07:45:47.080012718Z\ndeletion_time n/a\ndestroyed false\nversion
1\nKey Value\n--- -----\ncreated_time 2022-01-27T07:45:47.466863061Z\ndeletion_time n/a\ndestroyed
false\nversion 1\nKey Value\n--- -----\ncreated_time 2022-01-27T07:45:47.999533596Z\ndeletion_time
n/a\ndestroyed false\nversion 2", "stdout_lines": ["Key Value", "--- -----", "created_time
2022-01-27T07:45:47.080012718Z", "deletion_time n/a", "destroyed false", "version 1", "Key Value",
"--- -----", "created_time 2022-01-27T07:45:47.466863061Z", "deletion_time n/a", "destroyed false",
"version 1", "Key Value", "--- -----", "created_time 2022-01-27T07:45:47.999533596Z", "deletion_time
n/a", "destroyed false", "version 2"]}

See also the result in Vault UI:

To Reproduce Steps to reproduce the behavior: Try to deploy the network-cordav2.yaml network for Corda OS to Amazon EKS (v1.19).

Expected behavior The nodes are deployed and the NMS can start up without issues. It should be able to find its certificates.

Environment (please complete the following information):

• Amazon EKS with K9S V1.19
• Vault 1.7.2
• Main branch of Bevel repo

[vanderheijden86]

Fixed it by doing a fresh redeploy of everything.

[vanderheijden86]

Unfortunately, now I’m getting this error in the NMS and Doorman pods. I can’t access vault. Do you have any idea what could have gone wrong? Thanks in advance.

│ init-certificates Getting secrets from Vault Server                                                                                                                                                              │
│ init-certificates { "errors": [ "permission denied" ] }                                                                                                                                                          ││ init-certificates ERROR: unable to retrieve vault login token: {                                                                                                                                                 │
│ init-certificates   "errors": [                                                                                                                                                                                  ││ init-certificates     "permission denied"                                                                                                                                                                        │
│ init-certificates   ]                                                                                                                                                                                            ││ init-certificates }                                                                                                                                                                                              │
│ init-certificates stream closed