[Issue] Spring Cloud Config Server makes unnecessary calls to Vault

Hello Spring Cloud Team,

I wanted to raise an issue here on what seems to be a bug.

Setup: The setup is Spring Boot 2.5.1 + Ilford 2020.0.3 + Spring Cloud Config Server (This can be reproduced 100%, even without Spring Cloud Config Client) + Vault Server Backend to protect the secret + Actuator + Spring Boot Admin.

Issue, actual: On each call for /health endpoint (and other actuator endpoints), Spring Cloud Config Server will make (unnecessary) calls to Vault server.

Expected: I think Spring Cloud Config Server should not make calls to Vault server for /health and other non configuration endpoint.

The call to Vault should only happen when a Spring Cloud Config Client registers itself to the client to retrieve the config/secrets, or when a /refresh endpoint was invoked, not always.

Details: We observed this issue when our Vault instance was brought down. During investigation, we observed more than 2000 requests per minute were made against the Vault instance. Upon investigation, we believe every time a /health was called against Spring Cloud Config Server (by Kubernetes health probes) + other apps + when a Spring Cloud Config Client reports itself to Spring Boot admin, Spring Cloud Config Server will unnecessary flood the Vault server.

Logs:

2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] o.a.coyote.http11.Http11InputBuffer      : Received [GET /health HTTP/1.1
2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] o.a.c.authenticator.AuthenticatorBase    : Security checking request GET /health
2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] org.apache.catalina.realm.RealmBase      :   No applicable constraints defined
2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] o.a.c.authenticator.AuthenticatorBase    : Not subject to any constraint
2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] org.apache.tomcat.util.http.Parameters   : Set encoding to UTF-8
2021-06-20 13:23:10.731 DEBUG 3518 --- [nio-8989-exec-6] o.s.web.servlet.DispatcherServlet        : GET "/health", parameters={}
2021-06-20 13:23:10.732 DEBUG 3518 --- [nio-8989-exec-6] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped to Actuator web endpoint 'health'
2021-06-20 13:23:10.733 DEBUG 3518 --- [nio-8989-exec-6] o.s.web.client.RestTemplate              : HTTP GET https://vault.com:443/path/local,vault/data/app
2021-06-20 13:23:10.733 DEBUG 3518 --- [nio-8989-exec-6] o.s.web.client.RestTemplate              : Accept=[application/json, application/*+json
2021-06-20 13:23:10.733 DEBUG 3518 --- [nio-8989-exec-6] h.i.c.PoolingHttpClientConnectionManager : Connection request: [route: {s}->https://vault.com:443][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20]
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] o.a.http.impl.execchain.MainClientExec   : Executing request GET /path/local,vault/data/app HTTP/1.1
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] o.a.http.impl.execchain.MainClientExec   : Target auth state: UNCHALLENGED
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] o.a.http.impl.execchain.MainClientExec   : Proxy auth state: UNCHALLENGED
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> GET /path/local,vault/data/app HTTP/1.1
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> Accept: application/json, application/*+json
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> X-Vault-Token: token
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> Host: vault.com:443
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> Connection: Keep-Alive
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.13 (Java/11.0.6)
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.headers                  : http-outgoing-0 >> Accept-Encoding: gzip,deflate
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.wire                     : http-outgoing-0 >> "GET /path/local,vault/data/app HTTP/1.1[\r][\n]"
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.wire                     : http-outgoing-0 >> "Accept: application/json, application/*+json[\r][\n]"
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.wire                     : http-outgoing-0 >> "X-Vault-Token: token[\r][\n]"
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.wire                     : http-outgoing-0 >> "Host: vault.com:443[\r][\n]"
2021-06-20 13:23:10.735 DEBUG 3518 --- [nio-8989-exec-6] org.apache.http.wire                     : http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"

2021-06-20 13:23:10.903 DEBUG 3518 --- [nio-8989-exec-6] o.s.web.client.RestTemplate              : HTTP GET https://vault.com:443/path/local,vault/data/application

Reproducible project: Please find this link to a simple 4 files only 100% reproducible project.

https://github.com/patpatpat123/vaulconfigserverissue

Spring Cloud Config Server Team, do you mind helping to check why the server will make all those calls to Vault please, especially upon /health or /instances invocation? Is there any workaround now to reduce all those unnecessary calls?

Thank you

[ryanjbaxter]

@spencergibb that is exactly what I was thinking

[mp911de]

Spring Vault provides health check methods, see VaultSysOperations.health().