Cannot renew a MongoDB lease due to lease expired

How To Reproduce

• Vault Server Configuration

1. Start vault server in dev mode vault server -dev

2. Enable app-role vault auth enable approle

3. Import a policy vault write sys/policy/demo-policy policy=@demo-policy.hcl (the file is listed later)

4. Create an app role with unlimited tokens and secret ids vault write auth/approle/role/readwrite secret_id_ttl=100m token_num_uses=0 token_ttl=100m token_max_ttl=100m secret_id_num_uses=0 policies="default,demo-policy"

5. Enable database secrets vault secrets enable database

6. Configure MongoDB connection info. vault write database/config/vault-mongodb-demo-database plugin_name=mongodb-database-plugin allowed_roles="readwrite" connection_url="mongodb://{{username}}:{{password}}@[myip]/admin?ssl=false" username="[myusername]" password="[mypassword]"

7. Configure a role that maps a name vault write database/roles/readwrite db_name=vault-mongodb-demo-database creation_statements='{ "db": "vaultdemo", "roles": [{ "role": "dbOwner" }, {"role": "readWrite", "db": "vaultdemo"}] }' default_ttl="1m" max_ttl="1m"

8. Get roleid which will be the value of spring.cloud.vault.app-role.role-id vault read auth/approle/role/readwrite/role-id

9. Get secret id which will be the value of spring.cloud.vault.app-role.secret-id vault write -f auth/approle/role/readwrite/secret-id

• demo-policy.hcl file

path "secret/bootstrap" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "secret/application" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "database/creds/readwrite" {
	capabilities = ["create", "read", "update", "delete", "list"]
}
path "database/creds/readwrite/*" {
        capabilities = ["create", "read", "update", "delete", "list"]
}
path "sys/leases/*" {
	capabilities = ["create", "read", "update", "delete", "list"]
}

• bootstrap.yml file

spring:
  cloud:
    vault:
      host: localhost
      port: 8200
      scheme: http
      uri: http://localhost:8200
      connection-timeout: 5000
      read-timeout: 15000
      config:
        lifecycle:
          enabled: true
        order: -10
      authentication: APPROLE
      app-role:
        role-id: [from step 8]
        secret-id: [from step 9]
      mongodb:
        enabled: true
        role: readwrite
        backend: database
        username-property: mongodb.username
        password-property: mongodb.password

• Components version

1. Spring Boot 2.0.6.RELEASE

2. Spring Cloud Finchley.SR1

3. Spring Vault core 2.1.1.BUILD-SNAPSHOT

4. Vault v0.11.4

Error Description

After about 1 minute from starting the demo app, I got the following exception:

2018-10-25 22:11:26.616  WARN 5446 --- [g-Cloud-Vault-2] LeaseEventPublisher$LoggingErrorListener : [RequestedSecret [path='database/creds/readwrite', mode=RENEW]] Lease [leaseId='database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V', leaseDuration=PT10S, renewable=true] Cannot renew lease: lease expired

org.springframework.vault.VaultException: Cannot renew lease: lease expired
	at org.springframework.vault.core.lease.SecretLeaseContainer.doRenewLease(SecretLeaseContainer.java:604) [spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.vault.core.lease.SecretLeaseContainer.lambda$scheduleLeaseRenewal$0(SecretLeaseContainer.java:503) [spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.vault.core.lease.SecretLeaseContainer$LeaseRenewalScheduler$1.run(SecretLeaseContainer.java:776) ~[spring-vault-core-2.1.1.BUILD-20181010.071554-4.jar:2.1.1.BUILD-SNAPSHOT]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) ~[spring-context-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_152]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]

And error message also shows from vault server:

2018-10-25T22:11:26.607+0800 [ERROR] secrets.system.system_79b963f3: lease renewal failed: lease_id=database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V error="lease expired"
2018-10-25T22:11:26.690+0800 [INFO]  expiration: revoked lease: lease_id=database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V

Besides, before the exception, there’s some info of spring vault scheduling the renewal:

2018-10-25 22:10:26.506 DEBUG 5446 --- [           main] o.s.v.c.e.LeaseAwareVaultPropertySource  : Requesting secrets from Vault at database/creds/readwrite using RENEW
2018-10-25 22:10:26.558 DEBUG 5446 --- [           main] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:26.559 DEBUG 5446 --- [           main] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 60
2018-10-25 22:10:36.561 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:36.570 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:36.570 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 50
2018-10-25 22:10:46.574 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:46.578 DEBUG 5446 --- [g-Cloud-Vault-2] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:46.579 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 40
2018-10-25 22:10:56.579 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:10:56.585 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:10:56.585 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 30
2018-10-25 22:11:06.587 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:11:06.592 DEBUG 5446 --- [g-Cloud-Vault-2] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:11:06.592 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 20
2018-10-25 22:11:16.596 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
2018-10-25 22:11:16.600 DEBUG 5446 --- [g-Cloud-Vault-1] o.s.v.core.lease.SecretLeaseContainer    : Secret database/creds/readwrite with Lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V qualified for renewal
2018-10-25 22:11:16.600 DEBUG 5446 --- [g-Cloud-Vault-1] cretLeaseContainer$LeaseRenewalScheduler : Scheduling renewal for secret database/creds/readwrite with lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6V, lease duration 10
2018-10-25 22:11:26.603 DEBUG 5446 --- [g-Cloud-Vault-2] cretLeaseContainer$LeaseRenewalScheduler : Renewing lease database/creds/readwrite/RARStXyFlVRqfABSVobZUH6Vfor secret database/creds/readwrite
201

I’m using the version of 2.1.1.BUILD-SNAPSHOT because before the version I got 403 forbidden error when renewing the secret just as the question and the issue #255 described, and using the version according to the comment in #255 . However, it seems the problem reported in spring vault #319 still occurs in spring cloud vault.

[usr42]

@tonny1983 The follow-up post about how to rotate expiring relational Spring Cloud Vault database credentials without downtime is available: Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring

[usr42]

@tonny1983 I know that the thread is a bit old already. I hope it it is still helpful for you or other people stumbling across the same question. I’ve written a blog post about how to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault’s max_ttl: Hashicorp Vault max_ttl Killed My Spring App There will be a follow-up post which is more technology specific (HikariCP) but renews the PostrgreSQL credentials on the fly, which also works for really short-lived secrets (I’ve tested e.g. with max_ttl of 1 minute). If there is the demand I could also check how something similar could be achieved for MongoDB and then I would write another blog post with that information.