java.lang.InstantiationError: com.typesafe.scalalogging.Logger

Guys, I have odd problem that comes up once in a while and also magically disappears after I juggle with sbt clean comple, deleting target folder created by IDEA and restarting IntelliJ IDEA. Sometimes it takes me an hour to do these random actions which very very annoying.

I cannot find underlying cause.

What happens is I get following exception on start.

Exception in thread "main" java.lang.InstantiationError: com.typesafe.scalalogging.Logger
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:32)
    at com.typesafe.scalalogging.LazyLogging$class.logger(Logging.scala:28)
    at co.zzz.xxx.server.JournalsManager.logger$lzycompute(JournalsManager.scala:22)
    at co.zzz.xxx.server.JournalsManager.logger(JournalsManager.scala:22)
    at co.zzz.xxx.server.JournalsManager.<init>(JournalsManager.scala:42)
    at main$.main(main.scala:261)
    at main.main(main.scala)

where JournalsManager is object that extends com.typesafe.scalalogging.LazyLogging

In my build.scala I have following dependencies

          "com.typesafe.scala-logging" %% "scala-logging" % "3.5.0",
          "ch.qos.logback" % "logback-classic" % "1.1.+",

I’m on Scala 2.11.8

What should I do ?

[rbellamy]

@analytically - looks like the folks over at reactive-kafka figured it out - https://github.com/akka/reactive-kafka/issues/99.

Basically, if your dependency tree includes the older scala-logging-slf4j artifact you’ll see this exception.

Answer: exclude("com.typesafe.scala-logging", "scala-logging-slf4j_2.11")

[timo-aho]

I’ll leave my solution for Databricks logging problem for further reference.

tl;dr: Databricks uses old com.typesafe.scalalogging version 2.1.2. For newer ones, this is repaired.

Solution Scalalogging uses org.slf4j.Logger in the background. Let’s do our own Logger trait based on the latter:

package yourPackage

import org.slf4j.{Logger, LoggerFactory}

// Serializable logger
trait MyLogging {
  @transient
  protected lazy val logger: Logger =
    LoggerFactory.getLogger(getClass.getName)
}

Then let’s use this trait instead of com.typesafe.scalalogging.LazyLogging.

trait YourSparkApp extends yourPackage.MyLogging
{
  def main(argList: Array[String]): Unit = {
      implicit val logger: Logger = this.logger // For implicit usage.
      logger.info("This should work.")
      ...
    }
}

Longer explanation The currently newest Databricks 5.5 (Scala 2.11) uses old version of com.typesafe.scalalogging 2.1.2 as shown in https://docs.databricks.com/release-notes/runtime/5.5.html#installed-java-and-scala-libraries-scala-2-11-cluster-version To my knowledge, it is difficult to override this logger with other packages or versions in Spark. (I think there is an experimental flag for spark-submit.)

Logger should not be serialized, thus, it needs to be transient. However, 2.1.2 does not do this as shown here: 2.1.2: https://github.com/lightbend/scala-logging/blob/v2.1.2/scala-logging-slf4j/src/main/scala/com/typesafe/scalalogging/slf4j/Logging.scala#L28 As explained in this issue, in the newer versions, this is repaired. For instance 3.9.0: https://github.com/lightbend/scala-logging/blob/v3.9.0/src/main/scala/com/typesafe/scalalogging/Logging.scala#L27 Nevertheless, this currently does not help with Databricks because of the old library version used.

Hence, we need to create our own Logging trait based on org.sljf4j and inherit this (instead of com.typesafe.scalalogging.LazyLogging). We do not even need to change dependencies in Maven or SBT because org.slf4j is already included in the scalalogging library.