Better support for metrics in multi-services environments

Is your feature request related to a problem?

Described in https://github.com/elastic/apm/issues/65 - when service_name is not configured, the agent automatically discover service names. In servlet containers, this may result with multiple service names - one per webapp. However, metrics are still only visible in the context of the default servlet container service name (which is some generic name, depending on the traced framework).

Describe the solution you’d like

As described in the concluding comment of https://github.com/elastic/apm/issues/65, we are going to solve this by sending each metricset with all relevant service names.

With @tobiasstadler’s latest contributions to APM Server and the Java agent, we have everything in place to resolve this issue in this manner.

Some low-level guidelines:

• The metrics should not be sent to the default generic JVM service name anymore when there’s at least one service name override for a specific class loader (co.elastic.apm.agent.impl.ElasticApmTracer#serviceNameByClassLoader)
• This is all true for inherent agent metrics (JVM metrics and JMX). The way to decide which service names are relevant is to look into the class-loader–>service-name registry.
• For Micrometer metrics, handling should be a bit different (and better be handled in a separate PR):
  1. query the class-loader–>service-name map with the Micrometer classes class loader and if a service.name is found - use it
  2. query the class-loader–>service-name map with the context class loader of the thread doing the metric registration/update and if a service.name is found - use it
  3. If a mapped service.name was not found based on class loader lookup - send metrics to all service names in the map

Describe alternatives you’ve considered

The major alternative discussed is to send each metricset once with an array of all related service names. If the current approach turns out problematic, we can try this one instead.

Tasks

• elastic/apm-agent-java#2233
• elastic/apm-agent-java#1726

[felixbarny]

should we send the service.version as well within metricsets in order to consider this done?

Yes, I think so. #2233 is already a good step towards resolving this issue but I think it’s only really complete with #1726.

[tobiasstadler]

@eyalkoren Right now there is only one service.version per JVM, but I implemented sending the service.version in #1726 (when there can actually be more than one).

But you are right that right now the service.version will get lost if more than one service.name is “discovered”