Solr 8 JMX metrics enhancement

TL;DR

When Solr8 is using cloud sharding, some of the JMX metrics are not reporting. It would be great if they would!

About

If we look in the Solr7JmxValues.java class we can see that the `updateHandler’ is looking for beanName:

solr:dom1=core,dom2=*,category=UPDATE,scope=updateHandler,name=*

but when Solr is configured for sharding, the names are dynamically generated to an arbitrary depth, and may look more like this:

solr:dom1=core,dom2=solr-inventory,dom3=shard1,dom4=replica_n1,category=UPDATE,scope=updateHandler,name=adds

For caching, the NR Solr 7 jmx support is looking for

solr:dom1=core,dom2=*,category=CACHE,scope=searcher,name=documentCache

but Solr8 might look more like

solr:dom1=core,dom2=solr-inventory,dom3=shard1,dom4=replica_n1,category=CACHE,scope=searcher,name=documentCache

When the New Relic JMX component cannot find these beans, the data does not get reported and ends up showing up as zeros in the UI.

Feature Description

The agent should be enhanced to be able to find the sharded JMX beans. Rather than hard-coding a handful of fixed names, the agent should adapt to solr 8 and be able to list or otherwise enumerate the bean names and match the arbitrary-depth domain names, as shown above.

These beans should be queried/monitored by the agent and reported to New Relic for display in the NR1 UI.

Describe Alternatives

Solr uses dropwizard/codahale metrics internally, and so the New Relic dropwizard reporter might be able to be used to get the same telemetry. Some experimentation/exploration would be required to verify that the same information can be obtained…and also how it might map to a cohesive user experience.

Additional context

It is unknown what earliest version of Solr supports cloud/shard bean names with arbitrary domain depth. It is likely that this will continue in future versions of dropwizard.

Priority

“Really Want”. More than one customer has asked for this support.

[rahmnathan]

@mmulligan03 Really appreciate this! Using your configuration + query I’ve been able to get this stuff visualizing again, though obviously not as convenient as the built-in functionality that worked previously.

We’re considering Graphite and/or Prometheus as Solr is supposed to support those tools as well, but you’ve been immensely helpful (and prompt!) getting me past this issue.

[kford-newrelic]

@rahmnathan At the moment, we have a lot we want to accomplish for our agent roadmap and when we drew the line, Solr 8 didn’t make the cut. That doesn’t necessarily mean it’s a “forever” thing, just for the current roadmap. Of course, if there’s an enterprising engineer that wants to start with our existing instrumentation and create a PR with an update, that would be awesome!

@mmulligan03 really like your approach to crafting a custom instrumentation solution - we hope that others interested in Solr 8 can benefit from your hard work!