expose transfer bytes total (count) rather than just gauge of current value

Currently the worker Prometheus endpoint exposes transfer_incoming_bytes and transfer_outgoing_bytes as gauges—i.e., the current value at a single point in time.

A better way to expose this sort of data is as a monotonically increasing count metric type (this should be exposed as transfer_incoming_bytes_total and transfer_outgoing_bytes_total).

It’s easy to get rate from an accumulated count, but you can’t get accurate count from a sampled rate.

[ntabris]

I think we don’t expose actually bytes transferred over time, i.e., what @crusaderky calls “cumulative” values in https://github.com/dask/distributed/pull/6936#issuecomment-1230524443

That’s what I was asking for. If it’s not high-value, feel free to ignore for now though.

For context, host metrics can tell us how much data moves in/out of each worker. What it can’t exactly tell us (at least not easily) is how much of that is transfer vs data moving into/out of cluster (e.g., S3). I think it would be nice if Dask could tell us how many bytes of host network traffic is for transfer.

[gjoseph92]

I would also find this useful for benchmarking. Total amount of data transferred is a useful metric to compare when working on changes to scheduling.