Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Rebalance stuck, partitions rebalance not progressing in Kafka

See original GitHub issue

I am seeing an issue with the partition reassignments performed by CC: basically any rebalance that triggers more than one partition reassignment batch (tasks?) in kafka never completes. It gets stuck in the second batch. with cc waiting for reassignment to complete and kafka controller doing nothing. The reassign_partition zk node is being updated by CC but Kafka Controller is not triggering the rebalance. The only workaround is to delete /controller in kafka

Working with kafka 2.4.1 server side and a build of cc that has the 2.4.1 client libs (master)

Looking at https://github.com/linkedin/cruise-control/blob/b386141146dfe4cc013b8233c19434e09b50027c/cruise-control/src/main/scala/com/linkedin/kafka/cruisecontrol/executor/ExecutorUtils.scala#L89 it seems that CC is not using ReassignPartitionsCommand but instead updating the zk node directly. This apparently is not enough in newer version of Kafka.

Issue Analytics

State:
Created 3 years ago
Reactions:2
Comments:12 (5 by maintainers)

Top GitHub Comments

3reactions

isugimpycommented, Jun 23, 2020

A little late on this one since it’s closed, but running CC 2.4.8 on Kafka 2.4.0, I’m still experiencing this behavior. Would it be possible to reopen this issue, or should I open a new one?

1reaction

amurarucommented, Apr 23, 2020

One user pointed to https://issues.apache.org/jira/browse/KAFKA-9478 on gitter as the possible cause for this