Monitoring with Prometheus and Grafana

I’m looking into adding monitoring for pangeo, and would like to add a few metrics for Dask Gateway. I’m very new to prometheus and grafana, so apologies for any mistakes. Here are some rambling thoughts as I work through things. https://github.com/TomAugspurger/pangeo-cloud-federation/blob/metrics/gke/values.yaml has some helm configuration for the pangeo deployment I’m testing on.

I notice a couple of questions immediately:

1. Dask exposes prometheus metrics at scheduler-address/metrics, but Dask Gateway’s schedulers are ephemeral. How do we dynamically notify grafana / prometheus that it should add a datasource at an address? Should Dask-Gateway act as a broker here?
2. In my setup I’m using JupyterHub auth. Will all prometheus requests need to go through JupyterHub to access the dashboard?

I’ll continue to post here as I work through things.

cc @droctothorpe (who mentioned prometheus in https://github.com/dask/dask-gateway/pull/258) and @consideRatio

[TomAugspurger]

Hey, cool, that did it! Thanks!

Not the most impressive chart but it’s something 😃

For reference, I added the annotation manually with

kubectl -n metrics annotate service dask-gateway-9175cdffbe5c4c48bd469214522ca714 prometheus.io/scrape='true' prometheus.io/path='/metrics'

[TomAugspurger]

For reference, this was implemented for pangeo in https://github.com/pangeo-data/pangeo-cloud-federation/pull/626/ and https://github.com/pangeo-data/pangeo-cloud-federation/pull/648. The key bit was setting the prometheus.io/scrape: 'true' annotation on all the pods.