Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Version compatibility detection wrong in 2.1.0.BUILD-SNAPSHOT (Tue Apr 28 00:37:19 EEST 2015)

See original GitHub issue

I’m getting the following warning in my log.

15/04/29 10:04:38 WARN rdd.EsSpark: Incorrect classpath detected; Elasticsearch Spark compiled for Spark 1.0-1.2 but used with Spark 1.3.1

I checked the code and it’s using the absence of org.elasticsearch.spark.sql.EsSchemaRDDWriter to detect that this library was compiled for Spark 1.3.1.

In the snapshot I have that library does, indeed, exist.

Issue Analytics

State:
Created 8 years ago
Comments:8 (4 by maintainers)

Top GitHub Comments

1reaction

jbaieracommented, Aug 10, 2016

@fpopic - Good find! The compatibility check code assumes Spark 1.3 compatibility in the case that EsSchemaRDDWriter is absent. This class was only present in the old es-spark 1.0-1.2 compatibility packages. Since the 1.0-1.2 packages have been removed with the advent of Spark 2.0, the check now erroneously assumes that every version run is es-spark 1.3 compatibility. This is not a blocker, as it simply prints a warning log, but it is indeed confusing.

Could you kindly open a new Github issue for this so that we may track it separately from this currently closed one? Thanks.

0reactions

fpopiccommented, Aug 10, 2016

Thanks, I did it: https://github.com/elastic/elasticsearch-hadoop/issues/824 (it my first ever =) )