Proposal: resilience4j-micrometer integration should use tags

Context

Micrometer regards dimensions as a first-class citizen of time series and advises to not include them in the time series name (see https://micrometer.io/docs/concepts#_meters).

resilience4j-micrometer adds the name of CircuitBreakers (or Bulkheads, Retries and RateLimits for that matter) to the metric time series. This leads to limitations for querying (e.g. “find me all the open circuit breakers” would not be possible) and a large number of series being published (e.g. 6x #circuitbreakers) which is not optimal for most time series databases.

Proposal

With this issue I propose to use tags for dimensions for all the Metrics currently supported (CircuitBreaker, RateLimit, Bulkhead, Retry).

Consequences

The implementation changes would be trivial (see Gauge Fluent Builder) but would result in backward incompatible change in structure and names of time series. I am not sure if this is ok or if we should add this as a non-default option. I would be able to contribute a PR if we get agreement on an option.

Example

I use CircuitBreaker as an example here as it has the most complex structure of the four.

Current state

In Prometheus, this is what you get by default (cb1 being the circuitbreaker name, only showing one metric):

# HELP resilience4j_circuitbreaker_cb1_successful  
# TYPE resilience4j_circuitbreaker_cb1_successful gauge
resilience4j_circuitbreaker_cb1_successful 0.0

Proposed state: tags for names and metrics

To make discovery and drilldown easier, name and metric should be added as a dimension. This would allow drilldown on one specific circuitbreaker:

# HELP resilience4j_circuitbreaker  
# TYPE resilience4j_circuitbreaker gauge
resilience4j_circuitbreaker_stats{name="cb1",metric="successful"} 0.0
resilience4j_circuitbreaker_state{name="cb1"} 3.0

This version would support queries like:

• list of all open CircuitBreakers
• overview of buffered calls across all CircuitBreakers
• CircuitBreakers with more than 300 failed calls

[leonard84]

@RobWin I never meant that CircuitBreaker should keep a counter for all calls. What you described with the EventPublisher is close to what I had in mind. The only difference is that it should use tags instead of custom names and it should also pick up added CircuitBreakers.

[noroute]

@leonard84 is right. Dimensions of a metric (success/failure/id/etc.) should be rendered as label. Including dimensions in the name makes discovery of dimensions impossible.

In all cases I know, application identity is added by infrastructure (e.g. Prometheus relabeling) and does not need to be included by the app unless it has special knowledge about that identity.

With some further looking into the details of resilience4js submodules I found: resilience4j-prometheus already produces nice timeseries with tags:

# HELP resilience4j_circuitbreaker_calls Circuit Breaker Call Stats
# TYPE resilience4j_circuitbreaker_calls gauge
resilience4j_circuitbreaker_calls{name="backendB",call_result="successful",} 0.0

(see Code)

This also works nicely for the Spring Boot 1.x Starter since autoconfigure uses the Prometheus integration directly. The Spring Boot 2.x Starter (correctly) only supports autoconfigure for Micrometer.

If I’m not mistaken, the only thing that’s broken is resilience4j-micrometer since it forces all dimensions into the metric name, although Micrometer could easily handle dimensions and adapt the output to the respective time series database.

In short: use the names and tags from resilience4j-prometheus in resilience4j-micrometer and be done with it. This remains a breaking change (SemVer?) but should only affect people using Spring Boot 2 or Micrometer directly.