Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

V2.1.0 doesn't support spark 2.4.8

See original GitHub issue

Hi Spark Team, thanks for all the efforts, we use spark in GCP and are currently upgrading images due to log4j vulnerability, and currently facing an issue since we have the data we work with that doesn’t support spark 3.x and the current Google images use Spark 2.4.8, and after upgrading to the latest package v2.1.0, I got the error Exception in thread "main" java.lang.IllegalArgumentException: Unsupported spark version used: 2.4.8. Normalized spark version used: 2.4.8. Supported versions: 2.4.0, 2.4.4, 2.4.5, 2.4.1, 2.4.3 which I knew come from here, so after digging I found that it seem to be supported in https://github.com/dotnet/spark/blob/5f5215e6a4e011c4ece07a0f425ce01657ebcf6c/src/scala/microsoft-spark-2-4/src/main/scala/org/apache/spark/deploy/dotnet/DotnetRunner.scala but still I got that error when using microsoft-spark-2-4_2.11-2.1.0.jar.

Any idea why this happens?

Issue Analytics

State:
Created 2 years ago
Comments:13 (1 by maintainers)

Top GitHub Comments

1reaction

Niharikaduttacommented, Mar 2, 2022

@meywd Yes that is correct, if GCP is shipping Spark 2.4.8 with scala 2.12, then you would need to build microsoft-spark-2.4 against 2.12, along with any other jars you are using. Please change the scala.version in the microsoft-spark-2.4 project and refer to this doc on how to build from source.

0reactions

meywdcommented, Mar 4, 2022

Thanks, will do that

Top Results From Across the Web

Versioning Policy | Apache Spark

Versioning policy. Starting with Spark 1.0.0, the Spark project will follow the semantic versioning guidelines with a few deviations.

Overview - Spark 2.4.8 Documentation

This documentation is for Spark version 2.4.8. Spark uses Hadoop's client libraries for HDFS and YARN. Downloads are pre-packaged for a handful of...

Spark/Kafka error after updating Spark 2.4 to Spark 3.1

kafka-clients-2.8.0.jar. I striped my pyspark code to this that works with spark 2.4.1 but not with Spark 3.1.2:

Installation Requirements

Important: StreamSets does not provide support for Spark installations. The following table describes the features available with different Spark versions: ...

Azure Synapse Runtime for Apache Spark 2.4 (EOLA)

Azure Synapse Analytics supports multiple runtimes for Apache Spark. This document will cover the runtime components and versions for the Azure ...