Cruise control doesn't auto rebalance even with appropriate goals

Cruise control version: v2.0.59

I am testing out cruise control and noticed that cruise control doesn’t auto-rebalance when we scale up the brokers.

These are the steps I did:

1. Have a 3 node Kafka cluster with constant traffic of ~600msgs/second (just 8 topics overall)
2. Have cruise control running as well with self.healing turned on and anomaly.notifier.class=com.linkedin.kafka.cruisecontrol.detector.notifier.SelfHealingNotifier
3. Increase the number of brokers to 6

Expectation:

• Cruise control detects ‘ReplicaDistribution’ goal is violated, proposes replica movement and executes rebalance and things are automatically distributed among all the brokers (including new).

Reality:

• I do see proposals from cruise control but unless I manually run rebalance, the new brokers don’t get any partitions.

Optimization has 131 inter-broker replica(6689 MB) moves, 0 intra-broker replica(0 MB) moves and 36 leadership moves with a cluster model of 1 recent windows and 100.000% of the partitions covered.

Stats for ReplicaDistributionGoal(FIXED):
AVG:{cpu:       2.246 networkInbound:      25.123 networkOutbound:      25.108 disk:    2325.525 potentialNwOut:      75.306 replicas:94.33333333333333 leaderReplicas:35.0 topicReplicas:9.433333333333334}
MAX:{cpu:       3.100 networkInbound:      29.466 networkOutbound:      62.520 disk:    4651.177 potentialNwOut:      88.265 replicas:103 leaderReplicas:43 topicReplicas:33}
MIN:{cpu:       1.640 networkInbound:      20.783 networkOutbound:       4.437 disk:       0.000 potentialNwOut:      62.349 replicas:85 leaderReplicas:28 topicReplicas:0}
STD:{cpu:       0.486 networkInbound:       4.341 networkOutbound:      20.123 disk:    2325.522 potentialNwOut:      12.957 replicas:8.673074554171793 leaderReplicas:6.4807406984078595 topicReplicas:6.869221668645902

But the executor does nothing:

ExecutorState: {state: NO_TASK_IN_PROGRESS}
AnalyzerState: {isProposalReady: true, readyGoals: [NetworkInboundUsageDistributionGoal, CpuUsageDistributionGoal, PotentialNwOutGoal, NetworkInboundCapacityGoal, LeaderBytesInDistributionGoal, DiskCapacityGoal, ReplicaDistributionGoal, RackAwareGoal, TopicReplicaDistributionGoal, NetworkOutboundCapacityGoal, CpuCapacityGoal, DiskUsageDistributionGoal, NetworkOutboundUsageDistributionGoal, ReplicaCapacityGoal]}
AnomalyDetectorState: {selfHealingEnabled:[BROKER_FAILURE, DISK_FAILURE, GOAL_VIOLATION, METRIC_ANOMALY], selfHealingDisabled:[], selfHealingEnabledRatio:{BROKER_FAILURE=1.0, DISK_FAILURE=1.0, GOAL_VIOLATION=1.0, METRIC_ANOMALY=1.0}, recentGoalViolations:[], recentBrokerFailures:[], recentMetricAnomalies:[], recentDiskFailures:[], metrics:{meanTimeBetweenAnomalies:{GOAL_VIOLATION:0.00 milliseconds, BROKER_FAILURE:0.00 milliseconds, METRIC_ANOMALY:0.00 milliseconds}, meanTimeToStartFix:0.00 milliseconds, numSelfHealingStarted:0, ongoingAnomalyDuration=0.00 milliseconds}, ongoingSelfHealingAnomaly:None}

Questions:

• Is this expected? Is cruise control expected to auto-rebalance with self-healing turned on?
• Will self-healing turn on only when the brokers fail?
• Is there any other config knob that is missing which will help cluster auto-scale?

[efeg]

@mhaseebmlk This sounds like a configuration issue. Please see https://github.com/linkedin/cruise-control/issues/1270#issuecomment-659609862.

[mhaseebmlk]

@aravindvs were you able to get cruise control to trigger auto rebalance every time brokers are added while scaling up? I found this issue and also added ReplicaDistributionGoal to self.anomaly.detector.goals and noticed the cruise control triggered a cluster rebalance once. I then scaled down the cluster and scaled it back up again, however, this time, cruise control did not trigger an auto rebalance and the two new brokers did not get any partitions assigned to those.

I am wondering if this is an issue with cruise control or something wrong with my configuration. We would expect the auto rebalancing to get triggered every time the ReplicaDistributionGoal is violated but it looks like cruise control is not considering it being violated in this case.