question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to fetch the CPU usage metrics in elastic search for Openshift JVMs

See original GitHub issue

Dear Team,

we are unable to get the CPU metrics for our applications, we could see rest of all the other metrics, but not the CPU, this is a crucial parameter to assess the impact to application

Below are the details.

Elastic Search version: 7.17.1 Elastic APM Jar version: 1.30.0 Spring boot version: spring-boot-2.3.10.RELEASE OpenShift Master: v3.11.0+ec8630f-265 Kubernetes Master: v1.11.0+d4cacc0 OpenShift Web Console: v3.11.0+ea42280 Java: 11.0.14.1 (eclipse temurin)

ERROR/WARN:

2022-04-05 18:46:08,628 [main] WARN  co.elastic.apm.agent.impl.circuitbreaker.SystemCpuStressMonitor - Failed to obtain reference to the getSystemCpuLoad method of this JVM's OperatingSystemMXBean implementation
Screenshot 2022-04-06 at 12 39 42 AM

Originating stack trace

Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
2022-04-05 18:46:04,825 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.0 as link-api-gateway on Java 11.0.14.1 Runtime version: 11.0.14.1+1 VM version: 11.0.14.1+1 (Eclipse Adoptium) Linux 3.10.0-1062.18.1.el7.x86_64
2022-04-05 18:46:04,829 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'my-api-gateway' (source: Environment Variables)
2022-04-05 18:46:04,830 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - environment: 'DEV' (source: Environment Variables)
2022-04-05 18:46:04,830 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - enable_experimental_instrumentations: 'true' (source: Java System Properties)
2022-04-05 18:46:04,830 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://eapm-nonprod-dmz.xxxxxxx.com:443'  (source: Environment Variables)
2022-04-05 18:46:04,830 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.xxxxxx.my.api.gateway' (source: Environment Variables)
2022-04-05 18:46:08,628 [main] WARN  co.elastic.apm.agent.impl.circuitbreaker.SystemCpuStressMonitor - Failed to obtain reference to the getSystemCpuLoad method of this JVM's OperatingSystemMXBean implementation
2022-04-05 18:46:08,634 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-04-05 18:46:10,113 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-02-23T22:31:58Z",  "build_sha": "990c162bf1d5106be95d1430c6f3e98df865df7d",  "publish_ready": true,  "version": "7.17.1"}
2022-04-05 18:46:10,211 [elastic-apm-remote-config-poller] INFO  co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}
#logback.classic pattern: %d %-5level [%thread] %logger{0}: %msg%n
#logback.classic pattern: %d %-5level [%thread] %logger{0}: %msg%n
#logback.classic pattern: %d %-5level [%thread] %logger{0}: %msg%n

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v2.3.10.RELEASE)

Please let me know for more informatin.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

2reactions
sunilnagavellicommented, Apr 6, 2022

@jackshirazi , Marvellous!, This fixed the issue, Thank you so much!!!.

@eyalkoren and @hakansel , Issue is resolved now by adding jdk.management to our JRE Screenshot 2022-04-06 at 9 53 52 PM

Issue can be closed. Thank you.

1reaction
jackshirazicommented, Apr 6, 2022

Can you try adding jdk.management ?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting for Critical Alerts | Logging
Check if the Elasticsearch JVM Heap usage is high. oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_nodes/stats?pretty.
Read more >
Elasticsearch High CPU Usage - Main Causes and Solutions
High CPU is generally the consequence of JVM garbage collection, which in turn is caused by configuration or query related issues. In a...
Read more >
Chapter 12. Troubleshooting Logging OpenShift Container ...
Check if the Elasticsearch JVM Heap usage is high.
Read more >
How to Collect Elasticsearch Metrics - Datadog
There are five main HTTP APIs that you can use to monitor Elasticsearch: Node Stats API; Cluster Stats API; Index Stats API; Cluster...
Read more >
Manage compute resources | Elastic Cloud on Kubernetes [2.5]
Starting with Elasticsearch 7.11, the heap size of the JVM is automatically ... The two relevant metrics are kubernetes.pod.cpu.usage.limit.pct for CPU, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found