New relic java agent 7.5.0 throws NullPointerException and freezes the application

Description

NullPointerException is thrown by new relic java agent, when version 7.5.0 is enabled for my web application. When I invoke an endpoint that does some processing, it freezes the application fatally. The only way out to recover from this error is to restart the pod. This is a blocker / show stopper so I downgraded to version 6.5.3. Version 7.4.3 also has similar problem. The stack trace is as follows

Exception in thread "sdk-async-response-17-0" java.lang.NullPointerException
at util.TokenAndRefUtils.setThreadTokenAndRefCount(TokenAndRefUtils.java:29)
at util.TokenAwareRunnable.run(TokenAwareRunnable.java:27)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Exception in thread "Thread-5980" java.lang.NullPointerException
at util.TokenAndRefUtils.setThreadTokenAndRefCount(TokenAndRefUtils.java:29)
at util.TokenAwareRunnable.run(TokenAwareRunnable.java:27)
at java.base/java.lang.Thread.run(Thread.java:832)
Exception in thread "sdk-async-response-14-0" Exception in thread "Thread-5981" java.lang.NullPointerException
at util.TokenAndRefUtils.setThreadTokenAndRefCount(TokenAndRefUtils.java:29)
at util.TokenAwareRunnable.run(TokenAwareRunnable.java:27)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
java.lang.NullPointerException
at util.TokenAndRefUtils.clearThreadTokenAndRefCountAndTxn(TokenAndRefUtils.java:36)
at util.TokenAwareRunnable.run(TokenAwareRunnable.java:32)
at java.base/java.lang.Thread.run(Thread.java:832)

Expected Behavior

The application continue to work flawlessly without any disruption or denial of service, caused by new relic agent. I was under the impression that https://github.com/newrelic/newrelic-java-agent/pull/530 would fix this problem but I am wrong.

Your Environment

New relic java agent version: 7.5.0 New relic logging version: 2.3.1 Java: OpenJDK version 11.0.12 JVM: OpenJDK 64-Bit Server VM Zulu11.50+19-CA Scala: 2.13.6 Akka: 2.6.14 Akka HTTP: 10.2.4 Cats: 2.6.1 Cats effect: 3.2.2 Logback: 1.2.3

Additional context

See https://github.com/newrelic/newrelic-java-agent/issues/544

[viswanath7]

My apologies for the late response. Here’s the sample code that you requested. This is from a controller that offers real health check by verifying database connectivity along with an actor system check. When this endpoint was invoked by k8s liveliness probe, the container froze in an irrecoverable failure; which could only be recoverred via a restart of pod.

import akka.actor.ActorSystem
import akka.http.scaladsl.model.StatusCodes._
import akka.http.scaladsl.server.Route
import cats.effect.unsafe.implicits.global
import com.typesafe.scalalogging.LazyLogging
import de.heikoseeberger.akkahttpcirce.FailFastCirceSupport._
import io.circe.generic.JsonCodec

@JsonCodec final case class HealthStatus(status: String)

object HealthRoute extends LazyLogging {
  import akka.http.scaladsl.server.Directives._

  def systemRoute(implicit actorSystem: ActorSystem,
                  dbSupport: DatabaseSupport): Route = (get & path("system")) { requestContext =>
    HealthCheckService
      .healthCheck(actorSystem,dbSupport)
      .fold(healthCheckSuccess => complete(healthCheckSuccess),
        healthCheckFailure => {
          logger error s"Health check failed for ${healthCheckFailure.headName}. " +
            s"Statuses: ${healthCheckFailure.statuses.map(hce => s"${hce.name} -> ${hce.status}").toVector.toList.mkString(", ")}."
          complete((ServiceUnavailable, healthCheckFailure))
        }
      )
      .unsafeToFuture()
      .flatMap(route => route(requestContext))
  }
  }

where HealthCheckService is as shown below

object HealthCheckService {

  import slick.jdbc.PostgresProfile.api._

  def healthCheck(implicit actorSystem: ActorSystem, dbSupport: DatabaseSupport): HealthCheck[IO] = {
    HealthCheck
      .ok[IO](
        BuildInfo.name, BuildInfo.toMap
        .filterNot(entry => entry._1.equalsIgnoreCase("name") || entry._1.equalsIgnoreCase("builtAtMillis"))
        .map(entry => (entry._1, entry._2.toString))
      )
      .withActorSystemCheck(actorSystem.uptime > 1, akka.actor.ActorSystem.Version, akka.http.Version.current.some)
      .withPostgresCheck(dbSupport.database.run(sql"SELECT 1;".as[Int]))
  }
}

Relevant snippet from newrelic.yml configuration file is as shown below

distributed_tracing:
    enabled: true
# Cross Application Tracing adds request and response headers to
  # external calls using supported HTTP libraries to provide better
  # performance data when calling applications monitored by other New Relic Agents.
  cross_application_tracer:

    # Set to false to disable cross application tracing.
    # Default is true.
    enabled: true

  # Thread profiler measures wall clock time, CPU time, and method call counts
  # in your application's threads as they run.
  # This feature is not available to Lite accounts and is automatically disabled.
  thread_profiler:

    # Set to false to disable the thread profiler.
    # Default is true.
    enabled: true

  # New Relic Real User Monitoring gives you insight into the performance real users are
  # experiencing with your website. This is accomplished by measuring the time it takes for
  # your users' browsers to download and render your web pages by injecting a small amount
  # of JavaScript code into the header and footer of each page.
  browser_monitoring:

    # By default the agent automatically inserts API calls in compiled JSPs to
    # inject the monitoring JavaScript into web pages. Not all rendering engines are supported.
    # See https://docs.newrelic.com/docs/agents/java-agent/instrumentation/new-relic-browser-java-agent#manual_instrumentation
    # for instructions to add these manually to your pages.
    # Set this attribute to false to turn off this behavior.
    auto_instrument: true

  class_transformer:
    # This instrumentation reports the name of the user principal returned from
    # HttpServletRequest.getUserPrincipal() when servlets and filters are invoked.
    com.newrelic.instrumentation.servlet-user:
      enabled: false

    com.newrelic.instrumentation.spring-aop-2:
      enabled: false

    # This instrumentation reports metrics for resultset operations.
    com.newrelic.instrumentation.jdbc-resultset:
      enabled: false

    # This instrumentation reports very wrong numbers for Mongo async driver, e.g. it treats Mongo connection
    # as single database transaction. Should be enabled when NewRelic agent will support Mongo async driver.
    com.newrelic.instrumentation.mongodb-3.7:
      enabled: false

    # Classes loaded by classloaders in this list will not be instrumented.
    # This is a useful optimization for runtimes which use classloaders to
    # load dynamic classes which the agent would not instrument.
    classloader_excludes:
      groovy.lang.GroovyClassLoader$InnerLoader,
      org.codehaus.groovy.runtime.callsite.CallSiteClassLoader,
      com.collaxa.cube.engine.deployment.BPELClassLoader,
      org.springframework.data.convert.ClassGeneratingEntityInstantiator$ObjectInstantiatorClassGenerator,
      org.mvel2.optimizers.impl.asm.ASMAccessorOptimizer$ContextClassLoader,
      gw.internal.gosu.compiler.SingleServingGosuClassLoader,
    

[richard-gibson]

fixed on #859