Error trying to install mmlspark

When trying to install mmlspark, I encountered an error and googling it does not seem to yield many results.

Some Info about my setup

• MMLSpark Version: v1.0.0rc1
• Spark Version: v2.4.3
• Spark Platform: Spark on Linux CentOS 7

The stack trace:

[root@emr-header-1 user]# pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/opt/apps/ecm/service/spark/2.4.3-1.2.0/package/spark-2.4.3-1.2.0-bin-hadoop2.8/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.ml.spark#mmlspark_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-c9c90838-fd3d-49eb-863e-9b900f348032;1.0
        confs: [default]
        found com.microsoft.ml.spark#mmlspark_2.11;1.0.0-rc1 in spark-list
        found org.scalactic#scalactic_2.11;3.0.5 in central
        found org.scala-lang#scala-reflect;2.11.12 in local-m2-cache
        found org.scalatest#scalatest_2.11;3.0.5 in central
        found org.scala-lang.modules#scala-xml_2.11;1.0.6 in local-m2-cache
        found io.spray#spray-json_2.11;1.3.2 in central
        found com.microsoft.cntk#cntk;2.4 in central
        found org.openpnp#opencv;3.2.0-1 in central
        found com.jcraft#jsch;0.1.54 in central
        found org.apache.httpcomponents#httpclient;4.5.6 in central
        found org.apache.httpcomponents#httpcore;4.4.10 in central
        found commons-logging#commons-logging;1.2 in central
        found commons-codec#commons-codec;1.10 in local-m2-cache
        found com.microsoft.ml.lightgbm#lightgbmlib;2.3.100 in central
        found com.github.vowpalwabbit#vw-jni;8.7.0.3 in central
:: resolution report :: resolve 454ms :: artifacts dl 17ms
        :: modules in use:
        com.github.vowpalwabbit#vw-jni;8.7.0.3 from central in [default]
        com.jcraft#jsch;0.1.54 from central in [default]
        com.microsoft.cntk#cntk;2.4 from central in [default]
        com.microsoft.ml.lightgbm#lightgbmlib;2.3.100 from central in [default]
        com.microsoft.ml.spark#mmlspark_2.11;1.0.0-rc1 from spark-list in [default]
        commons-codec#commons-codec;1.10 from local-m2-cache in [default]
        commons-logging#commons-logging;1.2 from central in [default]
        io.spray#spray-json_2.11;1.3.2 from central in [default]
        org.apache.httpcomponents#httpclient;4.5.6 from central in [default]
        org.apache.httpcomponents#httpcore;4.4.10 from central in [default]
        org.openpnp#opencv;3.2.0-1 from central in [default]
        org.scala-lang#scala-reflect;2.11.12 from local-m2-cache in [default]
        org.scala-lang.modules#scala-xml_2.11;1.0.6 from local-m2-cache in [default]
        org.scalactic#scalactic_2.11;3.0.5 from central in [default]
        org.scalatest#scalatest_2.11;3.0.5 from central in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   15  |   0   |   0   |   0   ||   15  |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                [NOT FOUND  ] org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle) (3ms)

        ==== local-m2-cache: tried

          file:/root/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar

                [NOT FOUND  ] commons-codec#commons-codec;1.10!commons-codec.jar (0ms)

        ==== local-m2-cache: tried

          file:/root/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle)

                :: commons-codec#commons-codec;1.10!commons-codec.jar

                ::::::::::::::::::::::::::::::::::::::::::::::



:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle), download failed: commons-codec#commons-codec;1.10!commons-codec.jar]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1308)
        at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:315)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
  File "/usr/lib/spark-current/python/pyspark/shell.py", line 38, in <module>
    SparkContext._ensure_initialized()
  File "/usr/lib/spark-current/python/pyspark/context.py", line 316, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/usr/lib/spark-current/python/pyspark/java_gateway.py", line 46, in launch_gateway
    return _launch_gateway(conf)
  File "/usr/lib/spark-current/python/pyspark/java_gateway.py", line 108, in _launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

Would gladly appreciate any pointers in tracing the issue.

[John-Farseer]

hello, I meet the same trouble. I have solved the problem ! you can try the follows:

pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1 --repositories=https://mmlspark.azureedge.net/maven

Hope my way works for you!!! hhhhhhhhh

[gramhagen]

In your config there are some livy parameters, are you running this using Apache Livy?

If so, to get that working I had to exclude some packages during the configuration (which was done within the notebook using configure magic)

%%configure -f
{
    "name": "mmlspark",
    "conf": {
        "spark.jars.packages": "com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1",
        "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.11,org.scalactic:scalactic_2.11,org.scalatest:scalatest_2.11"
    }
}