Investigate Signer mismatch issue for microsoft-spark-*.jar

Is your feature request related to a problem? Please describe. My company uses Spark on Nomad to handle deployment and running of our Spark programs. I was recently tasked with evaluating whether or not it would be possible to use .NET Spark in conjunction with Spark on Nomad. After making sure that I am able to run the basic Scala examples, I tried to switch from the example .jar to microsoft-spark-2.4.x-0.3.0.jar, but got errors from the driver program.

Our specific setup for Spark on Nomad is to use Docker images based off of hashicorp/spark-nomad running in cluster mode. My idea for using .NET Spark is to replace the --class org.apache.spark.examples.SparkPi with --class org.apache.spark.deploy.DotnetRunner, and local:/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar with local:/app/microsoft-spark-2.4.x-0.3.0.jar (we have patched versions of the docker image with Spark updated to 2.4.0). When I tried this, I immediately get this exception:

java.lang.SecurityException: class "org.apache.spark.deploy.DotnetRunner"'s signer information does not match signer information of other classes in the same package

This leads me to believe that it should be possible to get this setup running, but I don’t have enough knowledge of the JVM/Spark side of things to know where to begin. I also noticed that there is a separate worker used in the other deployment methods to translate the .NET code for the executors, I’m not sure how that should be done.

Describe the solution you’d like Ideally all I would have to do is replace --class org.apache.spark.examples.SparkPi with --class org.apache.spark.deploy.DotnetRunner, and local:/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar with local:/app/microsoft-spark-2.4.x-0.3.0.jar

Describe alternatives you’ve considered We do not have the option of using a different orchestrator, so we would have to use normal Spark with Scala instead of this library

[imback82]

Cool. I will handle changing the title.

[halfbro]

I built the jar from source, and I can confirm that this solved our issues.

Should I change the PR to a bug report and change the title to something more fitting (ie [BUG]: java.lang.SecurityException during startup)? Or just leave it as is?