Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1

See original GitHub issue

Command

pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1

returns an error.

unresolved dependencies
com.microsoft.ml.spark:mmlspark_2.11;1.0.0-rc1 not found

I’m not sure if this is relevant but there’s a semicolon instead of a colon in that error. This is with Python 3.7.3, Apache Maven 3.6.1, jdk1.8.0_221, spark-2.4.3-bin-hadoop2.7

Issue Analytics

State:
Created 4 years ago
Comments:7

Top GitHub Comments

1reaction

mhamilton723commented, Oct 26, 2019

Hey @pairwiserr and @candalfigomoro sorry for this, looks like sbt thought it was a snapshot because of the -rc. You can get around this for now by using our maven repo:

by putting this in your build.sbt

resolvers += "MMLSpark Repo" at "https://mmlspark.azureedge.net/maven"

or adding that url to your spark.jars.repositories https://mmlspark.azureedge.net/maven

to your spark settings

0reactions

wangyumcommented, Sep 1, 2021

bin/spark-shell --conf spark.jars.repositories=https://mmlspark.azureedge.net/maven --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3

Top Results From Across the Web

packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1 #715

Command pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1 returns an error. unresolved dependencies ...

Microsoft Machine Learning for Apache Spark - NET

MMLSpark's API spans Scala, Python, Java, and R so you can integrate with any ecosystem. Try our PySpark Examples. Install. Spark Packages. Databricks....

how do I install parckage(such as mmlspark) to CDH cluster ...

pyspark --packages com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc1 ... I downloaded all packages and copy to local CDH cluster ...

PySpark 3.3.1 documentation - Apache Spark

PySpark supports most of Spark's features such as Spark SQL, DataFrame, Streaming, MLlib (Machine Learning) and Spark Core. PySpark Components. Spark SQL and ......

Spark Library Management - SQL Server Big Data Clusters

Scala Spark (Scala 2.12) and Hadoop base packages. PySpark (Python 3.8). Pandas, Sklearn, Numpy, and other data processing and machine learning ...