Error trying to install mmlspark
See original GitHub issueWhen trying to install mmlspark, I encountered an error and googling it does not seem to yield many results.
Some Info about my setup
- MMLSpark Version: v1.0.0rc1
- Spark Version: v2.4.3
- Spark Platform: Spark on Linux CentOS 7
The stack trace:
[root@emr-header-1 user]# pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/opt/apps/ecm/service/spark/2.4.3-1.2.0/package/spark-2.4.3-1.2.0-bin-hadoop2.8/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.ml.spark#mmlspark_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-c9c90838-fd3d-49eb-863e-9b900f348032;1.0
confs: [default]
found com.microsoft.ml.spark#mmlspark_2.11;1.0.0-rc1 in spark-list
found org.scalactic#scalactic_2.11;3.0.5 in central
found org.scala-lang#scala-reflect;2.11.12 in local-m2-cache
found org.scalatest#scalatest_2.11;3.0.5 in central
found org.scala-lang.modules#scala-xml_2.11;1.0.6 in local-m2-cache
found io.spray#spray-json_2.11;1.3.2 in central
found com.microsoft.cntk#cntk;2.4 in central
found org.openpnp#opencv;3.2.0-1 in central
found com.jcraft#jsch;0.1.54 in central
found org.apache.httpcomponents#httpclient;4.5.6 in central
found org.apache.httpcomponents#httpcore;4.4.10 in central
found commons-logging#commons-logging;1.2 in central
found commons-codec#commons-codec;1.10 in local-m2-cache
found com.microsoft.ml.lightgbm#lightgbmlib;2.3.100 in central
found com.github.vowpalwabbit#vw-jni;8.7.0.3 in central
:: resolution report :: resolve 454ms :: artifacts dl 17ms
:: modules in use:
com.github.vowpalwabbit#vw-jni;8.7.0.3 from central in [default]
com.jcraft#jsch;0.1.54 from central in [default]
com.microsoft.cntk#cntk;2.4 from central in [default]
com.microsoft.ml.lightgbm#lightgbmlib;2.3.100 from central in [default]
com.microsoft.ml.spark#mmlspark_2.11;1.0.0-rc1 from spark-list in [default]
commons-codec#commons-codec;1.10 from local-m2-cache in [default]
commons-logging#commons-logging;1.2 from central in [default]
io.spray#spray-json_2.11;1.3.2 from central in [default]
org.apache.httpcomponents#httpclient;4.5.6 from central in [default]
org.apache.httpcomponents#httpcore;4.4.10 from central in [default]
org.openpnp#opencv;3.2.0-1 from central in [default]
org.scala-lang#scala-reflect;2.11.12 from local-m2-cache in [default]
org.scala-lang.modules#scala-xml_2.11;1.0.6 from local-m2-cache in [default]
org.scalactic#scalactic_2.11;3.0.5 from central in [default]
org.scalatest#scalatest_2.11;3.0.5 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 15 | 0 | 0 | 0 || 15 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
[NOT FOUND ] org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle) (3ms)
==== local-m2-cache: tried
file:/root/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.6/scala-xml_2.11-1.0.6.jar
[NOT FOUND ] commons-codec#commons-codec;1.10!commons-codec.jar (0ms)
==== local-m2-cache: tried
file:/root/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle)
:: commons-codec#commons-codec;1.10!commons-codec.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.scala-lang.modules#scala-xml_2.11;1.0.6!scala-xml_2.11.jar(bundle), download failed: commons-codec#commons-codec;1.10!commons-codec.jar]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1308)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:315)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
File "/usr/lib/spark-current/python/pyspark/shell.py", line 38, in <module>
SparkContext._ensure_initialized()
File "/usr/lib/spark-current/python/pyspark/context.py", line 316, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/usr/lib/spark-current/python/pyspark/java_gateway.py", line 46, in launch_gateway
return _launch_gateway(conf)
File "/usr/lib/spark-current/python/pyspark/java_gateway.py", line 108, in _launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
Would gladly appreciate any pointers in tracing the issue.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:7 (2 by maintainers)
Top Results From Across the Web
Error trying to install mmlspark · Issue #771 - GitHub
When trying to install mmlspark, I encountered an error and googling it does not seem to yield many results. Some Info about my...
Read more >How to install mmlspark - pyspark - Stack Overflow
I am trying to deploy a mmlspark.lightgbm model on my pyspark code. I tried pip install mmlspark on master's ssh and I got...
Read more >Microsoft/MMLSpark - Gitter
Hi everyone, we are trying to perform offline install of Synapse ML 0.9.5 on Spark and having issue with "'JavaPackage' object is not...
Read more >Microsoft Machine Learning for Apache Spark - NET
MMLSpark's API spans Scala, Python, Java, and R so you can integrate with any ecosystem. Try our PySpark Examples. Install. Spark Packages. Databricks....
Read more >Installing Apache PySpark on Windows 10
This error message does not prevent the PySpark shell from starting. However if you try to run a standalone Python script using the...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
hello, I meet the same trouble. I have solved the problem ! you can try the follows:
pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1 --repositories=https://mmlspark.azureedge.net/maven
Hope my way works for you!!! hhhhhhhhh
In your config there are some livy parameters, are you running this using Apache Livy?
If so, to get that working I had to exclude some packages during the configuration (which was done within the notebook using configure magic)