Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cruise control w/ default configuration won't generate a proposal.

See original GitHub issue

Describe the bug I’ve deployed a Kafka crd with the default cruiseControl configuration. Cruise Control starts, but doesn’t generate proposals.

To Reproduce Steps to reproduce the behavior:

Deploy Kafka object
Deploy KafkaRebalance object
Describe KafkaRebalance object
Observe that Status.Conditions.Type=PendingProposal
Observe that the cruise control pod has errors

operationLogger:742 - Task [e2feb1d4-e0f7-45df-8c20-c8f971948701] calculation fails, exception:
java.util.concurrent.ExecutionException: Operation 'Rebalance' received exception. com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: com.linkedin.cruisecontrol.exception.NotEnoughValidWindowsException: There is no window available in range [-1, 1657129787915] (index [1, -1]). Wind
[report-06-07-2022_15-19-51.zip](https://github.com/strimzi/strimzi-kafka-operator/files/9058096/report-06-07-2022_15-19-51.zip)
ow index (current: 0, oldest: 0).

Expected behavior A KafkaRebalance object should have a rebalance proposal generated.

Environment (please complete the following information):

Strimzi version: 0.29.0
Installation method: Helm chart strimzi-kafka-operator-0.29.0
Kubernetes cluster: Kubernetes 1.21.12
Infrastructure: Self hosted cluster w/ 6 nodes tainted for exclusive use by Kafka

YAML files and logs attached

Issue Analytics

State:
Created a year ago
Comments:14 (8 by maintainers)

Top GitHub Comments

1reaction

kyguycommented, Jul 21, 2022

based on the link you provided, I had a coworker reboot all the kafka nodes with systemd.unified_cgroup_hierarchy=1 set. That resolved the problem. I can annotate a KafkaRebalance to refresh and approve a proposal. Then, CC executes the proposal properly.

Glad it is working now @benmathews! So from what understand this, the solution was to configure the Kubernetes nodes to use the same cgroups version either cgroups v1 by setting systemd.unified_cgroup_hierarchy=0 or cgroups v2 by setting systemd.unified_cgroup_hierarchy=1.

The only lingering question before closing this ticket, is if something could be added to strimzi to detect and warn of this problem?

Unfortunately, environment issues like this are not something that Strimzi can easily detect. That being said, the updates that are coming with OpenJDK 11.0.16 may have been able to help mitigate this particular issue but the cluster would still have been vulnerable to other cgroup issues

[1] https://bugs.openjdk.org/browse/JDK-8230305

0reactions

benmathewscommented, Jul 21, 2022

@kyguy thanks a lot for walking me through diagnosing and resolving this.