[Broker] Add a prometheus metric to indicate if a Bookie has been Quarantined

Is your feature request related to a problem? Please describe. Recently two of our 9 bookies were quarantined and we did not know until we saw logs in the Broker.

platform-pulsar-broker-1 platform-pulsar-broker 16:53:05.284 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - 
Bookie platform-pulsar-bookkeeper-1.platform-pulsar-bookkeeper.cogito-load.svc.cluster.local:3181 has been quarantined because of read/write errors.
platform-pulsar-broker-1 platform-pulsar-broker 16:53:05.284 [BookKeeperClientScheduler-OrderedScheduler-0-0] WARN  org.apache.bookkeeper.client.BookieWatcherImpl - 
Bookie platform-pulsar-bookkeeper-8.platform-pulsar-bookkeeper.cogito-load.svc.cluster.local:3181 has been quarantined because of read/write errors.

When these bookies are quarantined the overall throughput scalability of our system is reduced. We would like some scheme to monitor and alert if a bookie is quarantined.

Describe the solution you’d like We’d like to see some kind of Prometheus metrics added that indicated the number of Bookies that are currently quarantined. Once that metrics is added we can use it in Prometheus alerting to notify us of the problem.

Describe alternatives you’ve considered Can’t think of any

[hangc0276]

Hi @frankjkelly , we have already added the bookie quarantined metrics, but it doesn’t expose by default, because it is a bookie client metric. The metric name is pulsar_managedLedger_client_bookkeeper_client_BOOKIE_QUARANTINE and the type is Counter You can use the following PromQL to get metrics:

sum(irate(pulsar_managedLedger_client_bookkeeper_client_BOOKIE_QUARANTINE{job=$job_name, instance=~$instance_name}[1m])) by (instance)

If you want to expose this metric, you should turn on it in broker.conf

bookkeeperClientExposeStatsToPrometheus=true

[mattisonchao]

I think i can help on it ~