Bulk helper semaphore handling results in hanging await for long-running requests (exceeding flushInterval)

🐛 Bug Report

Bulk helper hangs forever when flushInterval is exceeded while iterator is already awaiting semaphore.

To Reproduce

Steps to reproduce the behavior:

1. Run below code against a test cluster - should complete successfully
2. Simulate long running server-side operation exceeding configured flushInterval. Multiple ways to do this but one way is to modify the compiled Helpers.js with the following:

client.bulk(Object.assign({}, bulkOptions, { body: bulkBody }), reqOptions, async (err, { body }) => {
await new Promise(resolve => setTimeout(resolve, flushInterval));
if (err) return callback(err, null)

1. Re-run below code and watch the hanging await caused by onFlushTimeout() invoked on a payload already awaiting semaphore()

Paste your code here:

'use strict'
const util = require("util")
const { Readable } = require('stream');
const { Client } = require('@elastic/elasticsearch');

async function* generator() {
    let i = 0
    while (i < 10) {
        await new Promise(resolve => setTimeout(resolve, 1 * 1000));
        yield { i: i++ };
    }
};

const readableStream = Readable.from(generator());

const elasticClient = new Client({
    [TESTCLUSTER]
});

(async () => {
    const bulkHelper = elasticClient.helpers.bulk({
        flushBytes: 43,
        concurrency: 1,
        datasource: readableStream,
        onDocument(doc) {
            console.log(doc)
            return {
                index: { _index: 'semaphoretest' }
            }
        },
        onDrop(doc) {
            console.log(doc);
        }
    }).catch((err) => {
        console.error(err);
    });

    while (util.inspect(bulkHelper).includes('pending')) {
        await new Promise(resolve => setTimeout(resolve, 1 * 1000));
        console.log('...waiting');
    }

    console.log(await bulkHelper);
})();

Expected behavior

Bulk helper awaits gracefully for queued requests to complete, error, or timeout.

Paste the results here:

{ i: 0 }
...waiting
{ i: 1 }
...waiting
...waiting
...waiting
...[forever]

Your Environment

• node version: v14.17.6
• @elastic/elasticsearch version: >=7.15.0
• os: Linux

[delvedor]

Heya, I wrote this to reproduce the issue:

const dataset = [
  { user: 'jon', age: 23 },
  { user: 'arya', age: 18 },
  { user: 'tyrion', age: 39 }
]

test('issue #1562', async t => {
  async function handler (req, res) {
    console.log(req.url)
    setTimeout(() => {
      res.writeHead(200, { 'content-type': 'application/json' })
      res.end(JSON.stringify({ errors: false, items: [{}] }))
    }, 1000)
  }

  const [{ port }, server] = await buildServer(handler)
  const client = new Client({ node: `http://localhost:${port}` })

  async function * generator () {
    const data = dataset.slice()
    for (const doc of data) {
      await sleep(1000)
      yield doc
    }
  }

  const result = await client.helpers.bulk({
    datasource: Readable.from(generator()),
    flushBytes: 1,
    flushInterval: 1000,
    concurrency: 1,
    onDocument (doc) {
      return {
        index: { _index: 'test' }
      }
    },
    onDrop (doc) {
      t.fail('This should never be called')
    }
  })

  t.type(result.time, 'number')
  t.type(result.bytes, 'number')
  t.match(result, {
    total: 3,
    successful: 3,
    retry: 0,
    failed: 0,
    aborted: false
  })

  server.stop()
})

My observations:

• if the flushInterval is exactly the same as the server timeout, the third request is never being sent
• if the flushInterval is different from the server timeout (doesn’t matter if higher or lower), the code works as expected.

It happens both in v7 and v8. This is weird, we’ll investigate. Thank you for reporting!

[YoannMa]

(comment to avoid stale)