Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG]: Trying to follow the "Getting Started" guide step by step

See original GitHub issue

Describe the bug I was following the Getting Started guide step by step. When I execute the following:

C:\Users\j.shaer\source\repos\HelloSpark\HelloSpark\bin\Debug\netcoreapp2.1>spark-submit 
`--class org.apache.spark.deploy.DotnetRunner ` --master local ` microsoft-spark-2.4.x-0.1.0.jar ` HelloSpark

I get this:

Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/C:/Users/j.shaer/source/repos/HelloSpark/HelloSpark/bin/Debug/netcoreapp2.1/%60--class
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
        at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Issue Analytics

State:
Created 4 years ago
Comments:26 (10 by maintainers)

Top GitHub Comments

6reactions

amgadmadkourcommented, Nov 11, 2020

One temporary work-around to avoid seeing the spark temporary files error is to add the following two lines in the log4j.properties file of your spark installation:

log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.logger.org.apache.spark.SparkEnv=ERROR

Source: Stack Overflow

3reactions

GoEddiecommented, Apr 25, 2019

Just to help others - in your spark directory there is a conf directory - add those two lines to the “log4j.properties” file. If there is no “log4j.properties” there should be a “log4j.properties.template” - copy the .template and remove the “.template” then add those lines at the top and the error will be hidden