Saving bundle generates error: java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
See original GitHub issueIssue Description
After updating from mleap v0.11.0 to v0.13.0, I am getting the following exception when I try to save a Spark model:
A needed class was not found. This could be due to an error in your runpath. Missing class: com/google/protobuf/GeneratedMessageV3
java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
...
at ml.combust.bundle.json.JsonSupport$class.$init$(JsonSupport.scala:82)
at ml.combust.bundle.json.JsonSupport$.<init>(JsonSupport.scala:295)
at ml.combust.bundle.json.JsonSupport$.<clinit>(JsonSupport.scala)
at ml.combust.bundle.serializer.JsonFormatModelSerializer.write(ModelSerializer.scala:48)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply$mcV$sp(ModelSerializer.scala:89)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply(ModelSerializer.scala:89)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2$$anonfun$apply$1.apply(ModelSerializer.scala:89)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2.apply(ModelSerializer.scala:89)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$2.apply(ModelSerializer.scala:89)
at scala.util.Success.flatMap(Try.scala:231)
at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:88)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:85)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:81)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:81)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:34)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$writeNode$1.apply(GraphSerializer.scala:30)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.GraphSerializer.writeNode(GraphSerializer.scala:30)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21)
at ml.combust.bundle.serializer.GraphSerializer$$anonfun$write$2.apply(GraphSerializer.scala:21)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35)
at ml.combust.bundle.serializer.GraphSerializer.write(GraphSerializer.scala:20)
at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:21)
at org.apache.spark.ml.bundle.ops.PipelineOp$$anon$1.store(PipelineOp.scala:14)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:87)
at ml.combust.bundle.serializer.ModelSerializer$$anonfun$write$1.apply(ModelSerializer.scala:83)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.ModelSerializer.write(ModelSerializer.scala:83)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:85)
at ml.combust.bundle.serializer.NodeSerializer$$anonfun$write$1.apply(NodeSerializer.scala:81)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.NodeSerializer.write(NodeSerializer.scala:81)
at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:34)
at ml.combust.bundle.serializer.BundleSerializer$$anonfun$write$1.apply(BundleSerializer.scala:29)
at scala.util.Try$.apply(Try.scala:192)
at ml.combust.bundle.serializer.BundleSerializer.write(BundleSerializer.scala:29)
at ml.combust.bundle.BundleWriter.save(BundleWriter.scala:31)
at com.tfs.test.MleapPipelineTest$$anonfun$2.apply(MleapPipelineTest.scala:88)
...
This message is generated from the following code block:
val sbc = SparkBundleContext().withDataset(model.transform(smallDF))
for(bf <- managed(BundleFile(s"jar:file:${file.getPath}")))
{ model.writeBundle.save(bf)(sbc).get } //Line# 88 MleapPipelineTest.scala
I investigated this some more, and looks like, at the moment the following gets pulled in as external library: Maven: com.google.protobuf:protobuf-java:2.5.0
It also looks like the com/google/protobuf/GeneratedMessageV3 is available from protobuf-java 3.1.0 onwards (https://github.com/google/safe-html-types/issues/1). I am not able to figure out how this specific version (2.5.0) is getting pulled, and where I should update so that the correct version gets pulled.
Anyone else gets this error? Any pointers. Thanks.
Issue Analytics
- State:
- Created 4 years ago
- Comments:9 (4 by maintainers)
Top Results From Across the Web
How can I solve "java.lang.NoClassDefFoundError"?
I got this error due to setting FormLayout(JGoodies) for layout setting of a Form. Which means there are issues with that class library...
Read more >3 ways to solve java.lang.NoClassDefFoundError in Java J2EE
I know how frustrating is to see Exception in thread "main" java.lang.NoClassDefFoundError, which is a manifestation of NoClassDefFoundError in Java.
Read more >"ERROR Unable to create Lookup for bundle java.lang ...
Ask Data process is down. Error: Could not find or load main class org.elasticsearch.common.settings.KeyStoreCli. Caused by: java.lang.
Read more >How to Resolve the NoClassDefFoundError in Java - Rollbar
NoClassDefFoundError is a Java error that occurs when the JVM is unable to find a class at runtime which was available at compile-time....
Read more >java.lang.NoClassDefFoundError. Does not work after adding ...
eclipse.osgi.util in, my bundle can not even be loaded with following errors. [AUDIT ] CWWKZ0404E: An exception was generated when trying to ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Okay, it’s spark-sql_2.11 dependency that’s actually bringing in the protobuf 2.5 dependency. I think you can just remove it entirely (I couldn’t really see where it was needed?) or alternatively you can include it like this in Maven.
With this change, most of the tests in CommonModelsSpec are passing.
Just a note on OneHotEncoder, because it’s deprecated in Spark 2.3 and will be removed from Spark soon, you should use OneHotEncoderEstimator instead. It works pretty similarly to the deprecated one. MLeap 0.13.0 only has support for OneHotEncoderEstimator, older MLeap versions will have some support older versions of Spark and OneHotEncoder, but just updating to use OneHotEncoderEstimator should be okay.
Let me know if you have any questions, or if not, if it’s ok to close the issue!
Tried this out today. I had to exclude protobuf-java from spark-yarn as well, and then the correct version got pulled in through mleap, as shown below.
This takes care of the issue mentioned here. However, then I got stuck with #423, a completely unrelated issue. Will follow-up on that separately. Closing this issue.