Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Bulk indexing errrors

See original GitHub issue

We are doing bulk indexing about every 5 minutes. Previously using elasticsearch-js 8.2.0 and Elasticsearch 2.4 everything worked fine.
The output log we create looks like similar to this

2017-02-24 11:15:16: Index data ...
2017-02-24 11:15:16:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:21:     indexed 7500 2017-02-24 11:15:21: documents: 5273.976ms
2017-02-24 11:15:21:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:26:     indexed 7500 2017-02-24 11:15:26: documents: 4030.945ms
2017-02-24 11:15:26:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:29:     indexed 7500 2017-02-24 11:15:29: documents: 3200.260ms
2017-02-24 11:15:29:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:32:     indexed 7500 2017-02-24 11:15:32: documents: 2794.364ms
2017-02-24 11:15:32:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:34:     indexed 7500 2017-02-24 11:15:34: documents: 2760.464ms
2017-02-24 11:15:34:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:37:     indexed 7500 2017-02-24 11:15:37: documents: 2216.049ms
2017-02-24 11:15:37:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:39:     indexed 7500 2017-02-24 11:15:39: documents: 2716.405ms
2017-02-24 11:15:39:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:42:     indexed 7500 2017-02-24 11:15:42: documents: 2426.167ms
2017-02-24 11:15:42:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:44:     indexed 7500 2017-02-24 11:15:44: documents: 2372.236ms
2017-02-24 11:15:44:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:47:     indexed 7500 2017-02-24 11:15:47: documents: 2548.165ms
2017-02-24 11:15:47:   bulk request of 7500 on promokits_1487952787856:promokit...
2017-02-24 11:15:49:     indexed 7500 2017-02-24 11:15:49: documents: 2566.625ms
2017-02-24 11:15:49:   bulk request of 714 on promokits_1487952787856:promokit...
2017-02-24 11:15:49:     indexed 714 2017-02-24 11:15:49: documents: 323.010ms

After upgrading to elasticsearch-js 12.1.1 and Elasticsearch 5.2.0 we get random errors a few times a day.

2017-02-24 06:54:44: Index data ...
2017-02-24 06:54:44:   bulk request of 7500 on promokits_1487937219324:promokit...
2017-02-24 06:54:45: Elasticsearch ERROR: 2017-02-24T11:54:45Z
Error: Request error, retrying
  POST https://host1:9280/_bulk  => socket hang up
      at Log.error (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/transport.js:240:18)
      at HttpConnector.<anonymous> (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/connectors/http.js:162:7)
      at ClientRequest.wrapper (/etc/gigsalad_crons/es-indexer/node_modules/lodash/lodash.js:4968:19)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at TLSSocket.socketCloseListener (_http_client.js:284:9)
      at emitOne (events.js:101:20)
      at TLSSocket.emit (events.js:188:7)
      at TCP._handle.close [as _onclose] (net.js:492:12) 
2017-02-24 06:54:48:     indexed 7500 2017-02-24 06:54:48: documents: 4253.678ms
2017-02-24 06:54:48:   bulk request of 7500 on promokits_1487937219324:promokit...
...

or sometimes the error will be

Error: Request error, retrying
  POST https://host1:9280/_bulk  => write EPIPE
      at Log.error (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/log.js:225:56)
      at checkRespForFailure (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/transport.js:240:18)
      at HttpConnector.<anonymous> (/etc/gigsalad_crons/es-indexer/node_modules/elasticsearch/src/lib/connectors/http.js:162:7)
      at ClientRequest.wrapper (/etc/gigsalad_crons/es-indexer/node_modules/lodash/lodash.js:4968:19)
      at emitOne (events.js:96:13)
      at ClientRequest.emit (events.js:188:7)
      at TLSSocket.socketErrorListener (_http_client.js:308:9)
      at emitOne (events.js:96:13)
      at TLSSocket.emit (events.js:188:7)
      at onwriteError (_stream_writable.js:319:10)
      at onwrite (_stream_writable.js:337:5)
      at WritableState.onwrite (_stream_writable.js:89:5)
      at fireErrorCallbacks (net.js:462:13)
      at TLSSocket.Socket._destroy (net.js:503:3)
      at WriteWrap.afterWrite (net.js:795:10)

These errors always occur on the first bulk index request and the retry is always successful so everything does get indexed fine.

I would like to know why this error is occurring and how to prevent it.

Issue Analytics

State:
Created 7 years ago
Comments:12 (4 by maintainers)

Top GitHub Comments

5reactions

calvinwiebecommented, May 31, 2017

@spalger any guidance on this? Thanks!

2reactions

baronworkscommented, Nov 22, 2017

similar issues elasticsearch 5.6.4, elasticsearch-js 14.0.0 (as well as 13.3.1) and nodejs v8.9.1 everything works fine with 13.0.0-beta2 client config: { apiVersion: ‘5.x’, hosts: [‘localhost:9200’], log: [‘error’, ‘warning’], sniffOnStart: false } Working on upgrading from elasticsearch 2.4.3 with elasticsearch-js 11.0.1 and v6.11.1 where it was also working.