Aggregation of t-digest types broken by default

In practice, summary data in prometheus can be hard to aggregate, because they lose a bit of data. I’ve been using a custom IPC setup to push summary data generated on workers to a master’s summary metric to avoid integration. However, this is cumbersome and I’d like to use the default metrics aggregator provided by prom-client. Can you speak to how this works in practice? It’s a bit difficult for me to figure out.

[vincentwoo]

Ah, I think you are correct. Here’s a test script:

const cluster = require('cluster')
const Prometheus = require('prom-client')

if (cluster.isMaster) {
  for (let i = 0; i < 3; i++) {
    cluster.fork()
  }
  const metricsServer = require('express')()
  const aggregatorRegistry = new Prometheus.AggregatorRegistry()
  metricsServer.get('/metrics', function(req, res) {
    aggregatorRegistry.clusterMetrics((err, metrics) => {
      res.set('Content-Type', aggregatorRegistry.contentType)
      res.end(metrics)
    })
  })
  metricsServer.listen(8000)
} else {
  const summary = new Prometheus.Summary({
    name: 'test',
    help: 'test'
  })
  summary.observe(1)
}

And the output:

$ curl http://localhost:8000/metrics
# HELP test test
# TYPE test summary
test{quantile="0.01"} 3
test{quantile="0.05"} 3
test{quantile="0.5"} 3
test{quantile="0.9"} 3
test{quantile="0.95"} 3
test{quantile="0.99"} 3
test{quantile="0.999"} 3
test_sum 3
test_count 3

I think this is probably not the behavior people would expect with default aggregation. I believe that summaries and histograms used the t-digest primitive, which appears to be built to be combined. If so, it seems like we could send the t-digest data over when summaries need to be aggregated.

Alternatively, might it be better for the library to do something under-the-covers where the master worker holds the “real” in-memory instrument, and the worker nodes, when observing measurements, merely pass those measurements over IPC to the master?

[vincentwoo]

I renamed the title to be a bit more on the mark and am bumping the thread! Any news?