Force kill connection ?

Is there any way to kill connection to kafka. I have tried using disconnect but it can get in infinity loop if onBatch or onMessage will never complete processing of one single message (which could be triggered by other things/errors). As a solution to my use case i would like to kill kafka connection and reestablish that again to reread uncommitted messages. Unfortunatle when i call disconnect i can some thing like:

[34mdebug:  timestamp=2019-12-02T08:36:45.211Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5000ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:36:46.212Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:47.214Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:48.219Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:49.220Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5003ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:36:50.220Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:51.220Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:52.223Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:53.231Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:54.235Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5001ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:36:55.238Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:56.241Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:57.244Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:58.249Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:36:59.251Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5003ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:37:00.252Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:01.253Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:02.254Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:03.256Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:04.260Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5003ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:37:05.264Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:06.265Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:07.269Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:08.270Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:09.270Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5005ms (sent: 0 | received: 0)
debug:  timestamp=2019-12-02T08:37:10.274Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:11.277Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:12.279Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:13.281Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
debug:  timestamp=2019-12-02T08:37:14.282Z, logger=kafkajs, message=waiting for consumer to finish..., groupId=ABC, memberId=ABCDE
info: [54210] Elapse: 5003ms (sent: 0 | received: 0)

And it never stops

[JaapRood]

Beyond consumer.disconnect, there is no way to force the connection to stop.

As a solution to my use case i would like to kill kafka connection and reestablish that again to reread uncommitted messages.

I would strongly advise against this, as the error that triggered before puts your app in an undefined state. Even if you’d kill the connection and somehow get the consumer to stop consuming (not sure how that would work), you’d run into undefined behaviour as a result of being in an undefined state. That can lead to some really hard to explain bugs, data corruption, etc. The thing to do there is to let the process crash, log the error and restart the process.

Your best course of action here, would be to figure out why your consumer “gets stuck” processing a message. Perhaps we can be of help there, maybe with the help of some code that replicates the issue. If that contains too much private information, some psuedo code could help, some logs might, etc.

[JaapRood]

That’s basically our approach as well. We’re actually running with the default 30 seconds for sessionTimeout, as a bit of time for things to rebalance is not that big of a deal. We’ve had it 5 seconds before, that increased the amount of rebalances we saw, so actually increased the total time we weren’t processing and consumer lag somewhat.