DatadogMeterRegistry StackOverflowError
See original GitHub issueHi,
we receive endless Stackoverflow exceptions if the datadog endpoint is not available anymore for any reason or they have to restart the servers or something:
The message in DD lookes like this:
{"timestamp":"2019-03-18T23:47:21.187+00:00","message":"failed to send metrics to datadog","logger":"i.m.datadog.DatadogMeterRegistry","thread_name":"datadog-metrics-publisher","status":"WARN","stack_trace":"java.lang.StackOverflowError: null\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasN
I don’t see any logs other than this, no warning that metrics couldn’t be sent or any other exception.
If i look into DatadogMeterRegistry i just see only one line where “hasNext()” is used, maybe you can find the loop in your code.
We’r using “micrometer-registry-datadog-1.1.2.jar” which is shipped with spring-boot 2.1
Issue Analytics
- State:
- Created 5 years ago
- Comments:15 (6 by maintainers)
Top GitHub Comments
ok, we’ll wait for the next jdk-11 updates and see if we’ll have the problem again, we had the problem only 2 times in the past 4 months, maybe it will take a while until we see it again …
@olamelin thanks for you investigation @izeye you can close the ticket if you want to, i’ll open a new one if we see the error again after the jdk-11.0.3 release
Since my last comment I created a parallel image of our code that was based on a openjdk12 image. In order to see if the JDK issue was the cause. The one using openjdk12 has not reported the stack overflow the last 6 days but the parallel image (which was restarted) has shown this error multiple times. In other words I think it is safe to say that this issue is caused by the bug described here https://bugs.openjdk.java.net/browse/JDK-8214129