Broker /health endpoint returns 200 OK when process first starts up

After restarting a broker process, we noticed our brokers have their /health endpoint return 200s right after the process starts up. Queries sent to the broker during this time return with BrokerResourceMissingError in the exceptions field. This happens for several seconds and then the /health endpoint will return 503 until it’s done building its routing maps.

We’re concerned that our load balances might pick up brokers that are being restarted and were wondering if it’s possible for them report 503 instead until it’s ready to serve queries for all tables it’s assigned to.

[jadami10]

My second question here is, can we make it configurable for service status to go to 503 once we send TERM or HUP. Otherwise even though the broker will go down after draining requests, the load balancer will still think it’s safe to send requests that way.

[xiangfu0]

For shutdown phase, both PinotServiceManager and Broker healthcheck should fail when you send the signal.

This is not the case as PinotServiceManager tries to stop the broker first, and the broker sleeps for default 10 seconds then just deregisters the handler. But #7880 seems to do all right here by just failing the PinotServiceManager instead. Are we comfortable rolling this out as the new default for everyone?

This PR enables health check when all bootstrap services are ready. When sending the signal, PinotServiceManager disable healthcheck first.

Thank you for doing this. This was my other thought at how to approach it, and it does cover all the services at once. I just have concerns that this approach is extremely prone to random deadlocks.

public static Status getServiceStatus() {
    return getServiceStatus(SERVICE_STATUS_CALLBACK);
  }

really needs to be limited in usage to only the /health endpoint. That said, I guess if you make it deadlock, then the integration tests fail, so maybe this is ok.

True, I’m making changes to not add PinotSM status check when the SM port < 0.