kafka-connect-elastic: ES 5 support

A sister issue reference: https://github.com/confluentinc/kafka-connect-elasticsearch/issues/55 I might be missing something, but on the outside, I’d like to see the two projects merged and properly maintained.

That being said, I’m getting: java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0] on the elastic side, when trying to use fast-data-dev/kafka-connect-elastic with docker.elastic.co/elasticsearch/elasticsearch:5.4.0.

[stheppi]

I will take care of the upgrade.This way it will be ‘properly’ maintained open source.

[costa]

@Antwnis Hey, I’ve pulled the latest image just now, and it seems to talk ES2 still. Also, I’m not sure how is it possible, but it both tries to talk ES2 protocol (according to the errors I see in the ES log) and shows the following error in its kafka-connect-ui page:

NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{172.20.0.3}{172.20.0.3:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
	at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226)
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.create(AbstractClient.java:1351)
	at com.sksamuel.elastic4s.CreateIndexDsl$CreateIndexDefinitionExecutable$$anonfun$apply$1.apply(CreateIndexDsl.scala:27)
	at com.sksamuel.elastic4s.CreateIndexDsl$CreateIndexDefinitionExecutable$$anonfun$apply$1.apply(CreateIndexDsl.scala:27)
	at com.sksamuel.elastic4s.Executable$class.injectFuture(Executable.scala:30)
	at com.sksamuel.elastic4s.CreateIndexDsl$CreateIndexDefinitionExecutable$.injectFuture(CreateIndexDsl.scala:24)
	at com.sksamuel.elastic4s.CreateIndexDsl$CreateIndexDefinitionExecutable$.apply(CreateIndexDsl.scala:27)
	at com.sksamuel.elastic4s.CreateIndexDsl$CreateIndexDefinitionExecutable$.apply(CreateIndexDsl.scala:24)
	at com.sksamuel.elastic4s.ElasticClient.execute(ElasticClient.scala:20)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticJsonWriter$$anonfun$createIndexes$1.apply(ElasticJsonWriter.scala:47)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticJsonWriter$$anonfun$createIndexes$1.apply(ElasticJsonWriter.scala:47)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticJsonWriter.createIndexes(ElasticJsonWriter.scala:47)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticJsonWriter.<init>(ElasticJsonWriter.scala:36)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticWriter$.apply(ElasticWriter.scala:43)
	at com.datamountaineer.streamreactor.connect.elastic.ElasticSinkTask.start(ElasticSinkTask.scala:59)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:231)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:145)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

while the specified ES node is actually available (checked the connectivity from inside the fast-data-dev container to that IP:port).