LightGBMRanker in example notebook throws `java.io.InvalidClassException: org.apache.spark.ml.Ranker`
See original GitHub issueDescribe the bug
I’m running the LightGBMRanker code (Cmds 37-41) in example notebook LightGBM - Overview.ipynb
from here on Azure Databricks. I get a bug in command lgbm_ranker_model = lgbm_ranker.fit(df)
(Cmd 41)
Info (please complete the following information):
- SynapseML Version: 2.12
- Spark Version 3.1.2
- Spark Platform Databricks DBR 9.1 LTS
- Installation co-ordinations:
com.microsoft.azure:synapseml_2.12:0.9.5-13-d1b51517-SNAPSHOT
Stacktrace
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 654.0 failed 4 times, most recent failure: Lost task 0.3 in stage 654.0 (TID 217219) (10.246.66.12 executor 72): java.io.InvalidClassException: org.apache.spark.ml.Ranker; local class incompatible: stream classdesc serialVersionUID = -3133757800595162776, local class serialVersionUID = 8149322387161021528
Issue Analytics
- State:
- Created a year ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
Saving Spark DataFrame as CSV gives ...
I've been using Spark lately and have started running into a problem when trying to save a Spark DataFrame. If I use the...
Read more >java.io.InvalidClassException:org.apache.spark.sql...
I am getting invalid class serialize exception spark submit client mode and it working good local mode.Can any - 220865.
Read more >Apache Spark Examples
These examples give a quick overview of the Spark API. Spark is built on the concept of distributed datasets, which contain arbitrary Java...
Read more >Launching visual recipes with Spark yields error : local ...
Hi,. the error implies that the jars on the images used by the kubernetes pods and the jars in the spark-submit called by...
Read more >Spark Connector 3.0.1 MongoSpark local class ...
I am using Apache Spark Java API and Mongo Spark Connector in PlayFramework 2.8.7 libraryDependencies ++= Seq( "org.apache.spark" ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Based on the fact you are running on AzureDatabricks I’m not really sure that is the issue. Wait, actually, can you show what libraries you have installed? I wonder if somehow you have two synapseml libraries installed in the databricks cluster at the same time? Perhaps ensuring you only have one version installed and restarting the cluster would fix this?
Yes, the uninstallation fixed the issue. Thank you!