Spring boot executable jar NoClassDefFoundError: Could not initialize class org.elasticsearch.hadoop.util.Version

What kind an issue is this?

• Bug report.

Issue description

I’m trying to run spark-to-es application. I created an application in spring boot and I was testing it in the IDE. I used gradle build to create a jar. But when i am trying to run jar, it has an error.

java -jar kafka-spark-es-spring-0.0.1-SNAPSHOT.jar

NoClassDefFoundError: Could not initialize class org.elasticsearch.hadoop.util.Version …

Steps to reproduce

Dependency:

implementation 'org.springframework.boot:spring-boot-starter'
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.4.3'
compile group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.4.3'
compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.4.3'
compile group: 'org.elasticsearch', name: 'elasticsearch-spark-20_2.11', version: '7.2.0'
...

Code: https://github.com/Hyunhoo-Kwon/realtime-data-pipeline/blob/spark-to-es-sample/kafka-spark-es-spring/src/main/java/com/study/kafkasparkesspring/job/WordCount.java

JavaStreamingContext streamingContext = new JavaStreamingContext(conf, Durations.seconds(10));
...
JavaPairDStream<String, ImmutableMap<String, Integer>> wordCountsMap = wordCounts.mapToPair(tuple2 -> new Tuple2<>(tuple2._1, ImmutableMap.of("count", tuple2._2)));
JavaEsSparkStreaming.saveToEsWithMeta(wordCountsMap, "spark", esConf);

error occurred on JavaEsSparkStreaming.saveToEsWithMeta

Strack trace:

Stack trace goes here
java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.hadoop.util.Version
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:577)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:65)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Version Info

OS: : Mac OS X JVM : 1.8.0_211 Hadoop/Spark: spark-2.4.3-hadoop-2.7 ES-Hadoop :
ES : elasticsearch-7.2.0

Feature description

[jbaiera]

It’s likely that you are affected by the same issue in #1358

[chris-gong]

What fixed this for me came from the duplicate issue linked in this thread, https://github.com/elastic/elasticsearch-hadoop/issues/1358#issuecomment-534808947, which was explicitly defining the elasticsearch-hadoop dependency in my pom file.

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-hadoop</artifactId>
    <version>7.17.2</version>
</dependency>

Replace 7.17.2 with whatever version you’re using.