Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
LightGBMRanker in example notebook throws `java.io.InvalidClassException: org.apache.spark.ml.Ranker`
See original GitHub issueDescribe the bug
I’m running the LightGBMRanker code (Cmds 37-41) in example notebook LightGBM - Overview.ipynb from here on Azure Databricks. I get a bug in command lgbm_ranker_model = lgbm_ranker.fit(df) (Cmd 41)
Info (please complete the following information):
- SynapseML Version: 2.12
- Spark Version 3.1.2
- Spark Platform Databricks DBR 9.1 LTS
- Installation co-ordinations:
com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT
Stacktrace
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 654.0 failed 4 times, most recent failure: Lost task 0.3 in stage 654.0 (TID 217219) (10.246.66.12 executor 72): java.io.InvalidClassException: org.apache.spark.ml.Ranker; local class incompatible: stream classdesc serialVersionUID = -3133757800595162776, local class serialVersionUID = 8149322387161021528
Issue Analytics
- State:
- Created a year ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
Saving Spark DataFrame as CSV gives ...
I've been using Spark lately and have started running into a problem when trying to save a Spark DataFrame. If I use the...
Read more >java.io.InvalidClassException:org.apache.spark.sql...
I am getting invalid class serialize exception spark submit client mode and it working good local mode.Can any - 220865.
Read more >Apache Spark Examples
These examples give a quick overview of the Spark API. Spark is built on the concept of distributed datasets, which contain arbitrary Java...
Read more >Launching visual recipes with Spark yields error : local ...
Hi,. the error implies that the jars on the images used by the kubernetes pods and the jars in the spark-submit called by...
Read more >Spark Connector 3.0.1 MongoSpark local class ...
I am using Apache Spark Java API and Mongo Spark Connector in PlayFramework 2.8.7 libraryDependencies ++= Seq( "org.apache.spark" ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Based on the fact you are running on AzureDatabricks I’m not really sure that is the issue. Wait, actually, can you show what libraries you have installed? I wonder if somehow you have two synapseml libraries installed in the databricks cluster at the same time? Perhaps ensuring you only have one version installed and restarting the cluster would fix this?
Yes, the uninstallation fixed the issue. Thank you!