Consumer takes 30 seconds for first message

Consumer code:

    await this.abisConsumer.connect()

    // Subscribe can be called several times
    await this.abisConsumer.subscribe({ topic: 'abis', fromBeginning: true })

    let startTime = performance.now()
    this.abisConsumer.run({
      eachMessage: async ({ /*topic, partition,*/ message }) => {
        console.log(`Took ${Math.round(performance.now() - startTime)} to execute`)
        console.log({
          key: message.key,
          value: message.value
        })
      },
    })  
    await this.abisConsumer.seek({ topic: 'abis', partition: 0, offset: 0 })

Only retrieving 6 messages and here is the output

Took 30827 to execute
{ key: null,
  value:
   <Buffer 7b 22 70 72 65 73 65 6e 74 22 3a 74 72 75 65 2c 22 62 6c 6f 63 6b 5f 6e 75 6d 22 3a 39 33 36 2c 22 61 63 63 6f 75 6e 74 22 3a 22 65 6f 73 69 6f 22 2c ... > }

Stuck on

{"level":"INFO","timestamp":"2019-02-05T21:21:27.618Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"deserialize-abis"}

The delay appears to be in joining the consumer group. However, the issue is only seen from kafkajs, works fine using command-line kafkacat

[tulios]

Hi @dmoena, ideally, you want to take advantage of automatic rebalances. When using manual assignment you end-up rebuilding most of what’s already in Kafka. I would say that a manual assignment should only be used in special cases.

The most common issue is when the group has to rebalance and the old consumers won’t acknowledge the rebalance, for example, the consumer fetches a big batch and have a long heartbeat configured, which means that the consumer won’t be aware of any changes in the group until it heartbeats or fetches the next batch. You can see early indications of this problem on your logs (Member <ID> in group test-group has failed, removing it from the group), this means that the group probably failed to heartbeat on a given interval and was kicked out of the group, triggering a new rebalance. This can indeed cause long rebalances but I would say Kafka is working as intended here.

[tulios]

Hi @jafri, are you using the same groupId of another consumer group? Because this time is influenced by the number of consumers in the group and their ability to stop processing and re-sync. Can you run your example with KAFKAJS_LOG_LEVEL=debug so we can see what’s happening, this process should be fast.

I run the example we have in the repo and it took some ms to join the group:

# Uncompressed
→ node examples/consumer.js
{"level":"INFO","timestamp":"2019-02-06T09:12:25.638Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}
{"level":"INFO","timestamp":"2019-02-06T09:12:25.725Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test-group","memberId":"example-consumer-7e704ac0-64d5-4fb2-b900-f29c54fcd332","leaderId":"example-consumer-7e704ac0-64d5-4fb2-b900-f29c54fcd332","isLeader":true,"memberAssignment":{"topic-test":[2,5,4,1,3,0]},"groupProtocol":"RoundRobinAssigner","duration":86}
Took 513 to execute

# Using Snappy
→ node examples/consumer.js
{"level":"INFO","timestamp":"2019-02-06T09:15:42.257Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}
{"level":"INFO","timestamp":"2019-02-06T09:15:42.309Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test-group","memberId":"example-consumer-e0931320-41e1-4b11-b4b7-cc2fdc308b8a","leaderId":"example-consumer-e0931320-41e1-4b11-b4b7-cc2fdc308b8a","isLeader":true,"memberAssignment":{"topic-test-2":[0]},"groupProtocol":"RoundRobinAssigner","duration":51}
Took 274 to execute

# 2nd run using Snappy
→ node examples/consumer.js
{"level":"INFO","timestamp":"2019-02-06T09:16:33.620Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"test-group"}
{"level":"INFO","timestamp":"2019-02-06T09:16:33.685Z","logger":"kafkajs","message":"[Runner] Consumer has joined the group","groupId":"test-group","memberId":"example-consumer-15246b24-092b-4ef1-bc2a-c1093fbb1a1f","leaderId":"example-consumer-15246b24-092b-4ef1-bc2a-c1093fbb1a1f","isLeader":true,"memberAssignment":{"topic-test-2":[0]},"groupProtocol":"RoundRobinAssigner","duration":64}
Took 721 to execute