Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PubSub to ElasticSearch Error

See original GitHub issue

I’m trying to ingest data to ES with the use of GCP Dataflow and the template PubSub to Elasticsearch. So far I’ve tried a couple of different deployments which are all on GCP using the trial option over at elastic.co.

The data I’m trying to ingest are metrics from our devices in a simple JSON format. The dataflow is configured by using the Cloud ID for my deployment and a custom UDF to format the data from PubSub. The type of data I leave default, so I’m not using the audit option as some guides sugest.

It all seems to go well, some metrics are ingested but then there is an error. view below:

Error message from worker: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:231)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1483)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1449)
java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:231)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1483)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1449)
java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:231)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1483)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1449)
java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:231)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1483)
        com.google.cloud.teleport.v2.elasticsearch.utils.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1449)

This Dataflow is using default configurations so I’d really expect this to work out of the box.

Issue Analytics

State:
Created 2 years ago
Comments:24

Top GitHub Comments

1reaction

felix-lessoercommented, Jun 15, 2022

@GustafKisi For me its working now with Elasticsearch 8.x !

1reaction

felix-lessoercommented, May 31, 2022

They are working on Elasticsearch 8.x support here: https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/369 Should be available soon within the UI.

Top Results From Across the Web

Pub/Sub to Elasticsearch Dataflow Template - GitHub

The Pub/Sub to Elasticsearch template is a streaming pipeline that reads messages from a Pub/Sub subscription and writes them to Elasticsearch as documents....

Sending message from pub/sub to Elasticsearch

I am trying to send messages from pub/sub subscriber to Elasticsearch index in queue (like row insert in SQL).

GCP Pub/Sub input | Filebeat Reference [8.5] - Elastic

Use the gcp-pubsub input to read messages from a Google Cloud Pub/Sub topic subscription. This input can, for example, be used to receive...

Pubsub input plugin problem - Logstash - Elastic Discuss

Hi, Trying to use https://github.com/Wondermall/logstash-input-google-cloud-pubsub I think I was able to configure all needed, ...

GCP Pub/Sub Subscription Creation | Elastic Security Solution ...

Elasticsearch is a trademark of Elasticsearch B.V., registered in the U.S. and in other countries. Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and...