Spark 3.1.1, scala 2.12 & xgboost 1.3.1 migration
See original GitHub issueHello mleap developers,
I’ve been busy migrating transmogrif.ai to use Spark 3.1.1, scala 2.12 and xgboost 1.3.1, and due to our heavy reliance on mleap I’ve had to play with the mleap master branch in order to make it work. I thought it would be a good time to share my findings cause we’ll ultimately need a new mleap build in Maven to complete the migration.
What I’ve done so far is:
- take the existing
spark-3.0.0
branch and rebase it on top of master - updated the spark (3.1.1), scala (2.12.10), akka (2.6.14 and 10.2.4 for akka-http) and xgboost (1.3.1) versions
- made a few modifications to the code to adapt to spark & xgboost API changes
All of which was enough to fix all of our JSON serialization/deserialization issues although we still have a number of unit tests to fix and we may discover more issues related to mleap. What I got so far is for sure not sufficient to release a new build but I’m wondering if/when you could help me make it happen. I’m happy to submit either a new PR, push a new branch or update the spark-3.0.0
branch with my changes. Let me know which you’d prefer.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:6
- Comments:11 (4 by maintainers)
Other PR seemed to have stalled, so I put together #765
I didn’t touch xgboost in #765, so still xgboost 1.0.0. xgboost 1.1.0 may or may not work – leaving the xgboost upgrade for its own PR.
Spark 2.4.x will not be compatible because spark 2.4 requires scala 2.11 while spark 3 requires scala 2.12. However, #765 does maintain backwards compatibility for deserializing a model trained on spark 2.4 into mleap runtime.