Saving bundle generates error: java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3

After updating from mleap v0.11.0 to v0.13.0, I am getting the following exception when I try to save a Spark model:

A needed class was not found. This could be due to an error in your runpath. Missing class: com/google/protobuf/GeneratedMessageV3
java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        ...
	at ml.combust.bundle.json.JsonSupport$class.$init$(JsonSupport.scala:82)
	at ml.combust.bundle.json.JsonSupport$.<init>(JsonSupport.scala:295)
	at ml.combust.bundle.json.JsonSupport$.<clinit>(JsonSupport.scala)
	at ml.combust.bundle.serializer.JsonFormatModelSerializer.write(ModelSerializer.scala:48)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply$mcV$sp(ModelSerializer.scala:89)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply(ModelSerializer.scala:89)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply(ModelSerializer.scala:89)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2.apply(ModelSerializer.scala:89)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2.apply(ModelSerializer.scala:89)
	at scala.util.Success.flatMap(Try.scala:231)
	at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:88)
	at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:85)
	at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:81)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:81)
	at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:34)
	at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:30)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.GraphSerializer.writeNode(GraphSerializer.scala:30)
	at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21)
	at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21)
	at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
	at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
	at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35)
	at ml.combust.bundle.serializer.GraphSerializer.write(GraphSerializer.scala:20)
	at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:21)
	at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:14)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:87)
	at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:83)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:83)
	at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:85)
	at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:81)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:81)
	at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:34)
	at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:29)
	at scala.util.Try$.apply(Try.scala:192)
	at ml.combust.bundle.serializer.BundleSerializer.write(BundleSerializer.scala:29)
	at ml.combust.bundle.BundleWriter.save(BundleWriter.scala:31)
	at com.tfs.test.MleapPipelineTest$$anonfun$2.apply(MleapPipelineTest.scala:88)
        ...

This message is generated from the following code block:

    val sbc = SparkBundleContext().withDataset(model.transform(smallDF))
    for(bf <- managed(BundleFile(s"jar:file:${file.getPath}")))
    {   model.writeBundle.save(bf)(sbc).get }    //Line# 88 MleapPipelineTest.scala

I investigated this some more, and looks like, at the moment the following gets pulled in as external library: Maven: com.google.protobuf:protobuf-java:2.5.0 It also looks like the com/google/protobuf/GeneratedMessageV3 is available from protobuf-java 3.1.0 onwards (https://github.com/google/safe-html-types/issues/1). I am not able to figure out how this specific version (2.5.0) is getting pulled, and where I should update so that the correct version gets pulled.

Anyone else gets this error? Any pointers. Thanks.

[ancasarb]

Okay, it’s spark-sql_2.11 dependency that’s actually bringing in the protobuf 2.5 dependency. I think you can just remove it entirely (I couldn’t really see where it was needed?) or alternatively you can include it like this in Maven.

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.3.0</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.protobuf</groupId>
                    <artifactId>protobuf-java</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

With this change, most of the tests in CommonModelsSpec are passing.

Just a note on OneHotEncoder, because it’s deprecated in Spark 2.3 and will be removed from Spark soon, you should use OneHotEncoderEstimator instead. It works pretty similarly to the deprecated one. MLeap 0.13.0 only has support for OneHotEncoderEstimator, older MLeap versions will have some support older versions of Spark and OneHotEncoder, but just updating to use OneHotEncoderEstimator should be okay.

Let me know if you have any questions, or if not, if it’s ok to close the issue!

[samikrc]

Tried this out today. I had to exclude protobuf-java from spark-yarn as well, and then the correct version got pulled in through mleap, as shown below.

try-mleap$ mvn dependency:tree -Dincludes=com.google.protobuf:protobuf-java
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building try-mleap 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ try-mleap ---
[INFO] com.tfs.test:try-mleap:jar:1.0-SNAPSHOT
[INFO] \- ml.combust.mleap:mleap-runtime_2.11:jar:0.13.0:compile
[INFO]    \- ml.combust.bundle:bundle-ml_2.11:jar:0.13.0:compile
[INFO]       \- com.google.protobuf:protobuf-java:jar:3.5.1:compile
[INFO] ------------------------------------------------------------------------

This takes care of the issue mentioned here. However, then I got stuck with #423, a completely unrelated issue. Will follow-up on that separately. Closing this issue.