Health and Liveness Actuator Connections Refused/Fail Under Heavy Load

Spring Boot 2.4.1

I’ve set up new Readiness and Liveness probes, which work nicely under normal conditions.

However, under heavy load, they return Liveness probe failed: Get http://<ip>:<port>/health/liveness: dial tcp <ip>:<port>: connect: connection refused to Kubernetes. Obviously, this is bad as it will then restart strained-but-generally-working containers.

Would really like a way to configure priority to management endpoints above normal traffic to make sure life/readiness checks don’t fail above all.

[wilkinsona]

Thanks. From what you’ve described thus far, I am skeptical that this is a Spring Boot problem. With the management server running on a separate port, its ability to accept connections will be unaffected by the load on the application endpoints as it uses a separate embedded server with a separate thread pool, etc. It may be possible for it to become unresponsive if the CPU is saturated to the extent that a thread in the management server doesn’t get scheduled to accept the connection. However, in that case, I think the behaviour you’re seeing is desirable as it would be just as likely to happen with a request to an application endpoint so the instance really isn’t alive and ready to handle requests.

Would really like a way to configure priority to management endpoints above normal traffic to make sure life/readiness checks don’t fail above all

I’m not sure that it would be possible for us to implement this, but even if it were, I think it would be a mistake to do so. If traffic to the management endpoints was prioritised you may get into a situation where an instance reports that it is alive and ready when it is, in fact, unable to handle any application traffic. It’s for this reason that we caution against configuring a separate management port as doing so can cause the same problem.

[wilkinsona]

Are you running Actuator on the same port as your application or have you configured management.server.port to make the Actuator available on a separate port?