Spark 3.1.1, scala 2.12 & xgboost 1.3.1 migration

Hello mleap developers,

I’ve been busy migrating transmogrif.ai to use Spark 3.1.1, scala 2.12 and xgboost 1.3.1, and due to our heavy reliance on mleap I’ve had to play with the mleap master branch in order to make it work. I thought it would be a good time to share my findings cause we’ll ultimately need a new mleap build in Maven to complete the migration.

What I’ve done so far is:

• take the existing spark-3.0.0 branch and rebase it on top of master
• updated the spark (3.1.1), scala (2.12.10), akka (2.6.14 and 10.2.4 for akka-http) and xgboost (1.3.1) versions
• made a few modifications to the code to adapt to spark & xgboost API changes

All of which was enough to fix all of our JSON serialization/deserialization issues although we still have a number of unit tests to fix and we may discover more issues related to mleap. What I got so far is for sure not sufficient to release a new build but I’m wondering if/when you could help me make it happen. I’m happy to submit either a new PR, push a new branch or update the spark-3.0.0 branch with my changes. Let me know which you’d prefer.

[jsleight]

@emitc2h @jsleight I was wondering is there a timeline by when we can provide Spark3 support?

Other PR seemed to have stalled, so I put together #765

[jsleight]

@jsleight would this mean that post fix mleap would remain backward compatible with xgboost 1.1.0 , Spark 2.4.x ?

I didn’t touch xgboost in #765, so still xgboost 1.0.0. xgboost 1.1.0 may or may not work – leaving the xgboost upgrade for its own PR.

Spark 2.4.x will not be compatible because spark 2.4 requires scala 2.11 while spark 3 requires scala 2.12. However, #765 does maintain backwards compatibility for deserializing a model trained on spark 2.4 into mleap runtime.