When using Promise.all to start connections, Kafka Broker times out

I had some code that essentially started two Kafka consumers (each for different topics), when doing:

await Promise.all([
  consumer1.start(),
  consumer2.start()
])

I’d receive errors (via logs) about the Kafka broker being unable to be connected to, but only for one of the consumers.

Finally, I changed it to:

await consumer1.start()
await consumer2.start()

And everything worked as expected.

The consumer start() code is essentially just:

  async start() {
    await this.consumer.connect();
    await this.consumer.subscribe({
      topic: this.topic,
      fromBeginning: true
    });

    await this.consumer.run({
       // ....
    })
  }

My gut feeling on this is that there’s some sort of race condition triggered in the Broker connection code when starting multiple consumers (or potentially multiple consumers and producers) in parallel.

[ThisIsMissEm]

Aha! Okay, yeah, increasing the connectionTimeout up to 10,000 helped, and the service started as expected. What’s a generally good amount of time to allow here? 1 second seems too low

On 28. Aug 2019, at 10:43, Túlio Ornelas [e-mail] wrote:

I can see that you are not using authentication in your test environment, so the authenticationTimeout won’t be used. It’s common for a single broker to delay things a bit, try to increase the connectionTimeout, the default value is 1s.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tulios/kafkajs/issues/480?email_source=notifications&email_token=AAAHQ242XKII625SERMOTJTQGY3BVA5CNFSM4IPGVLU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5KLLYA#issuecomment-525645280, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAHQ2Y4JQW4R45C3KACMP3QGY3BVANCNFSM4IPGVLUQ.

[JaapRood]

Having worked with time-outs before in previous deployments with other kafka clients: it should be less than the session timeout to prevent big issues, but network latency is definitely something to be tolerated too. We actually found 10s to be a good trade-off as well!