Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

System test failure on k8 cluster

See original GitHub issue

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug We are running the following regression test (RollingUpdateST-testKafkaAndZookeeperScaleUpScaleDown) and noticed the following issue on our k8 cluster (v1.18.5). When setting up the Persistent Storage for the Zookeeper nodes, it would set the storage claim to 1Mi causing the test to fail, as shown below.

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-my-cluster-2079133299-zookeeper-0   Bound    pvc-1cae6a93-3ec4-432a-bb50-d5a509f57619   1Mi        RWO            block-storage   18s
data-my-cluster-2079133299-zookeeper-1   Bound    pvc-482b1f8d-f240-474f-a17e-f17ca82482af   1Mi        RWO            block-storage   18s
data-my-cluster-2079133299-zookeeper-2   Bound    pvc-9a8d6b35-1afe-4a14-9df4-82099894d8a7   1Mi        RWO            block-storage   18s

Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Normal   Scheduled               <unknown>         default-scheduler        Successfully assigned namespace-0/my-cluster-2079133299-zookeeper-0 to worker1
  Normal   SuccessfulAttachVolume  82s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-1cae6a93-3ec4-432a-bb50-d5a509f57619"
  Warning  FailedMount             3s (x8 over 73s)  kubelet, worker1         MountVolume.MountDevice failed for volume "pvc-1cae6a93-3ec4-432a-bb50-d5a509f57619" : rpc error: code = Internal desc = signal: aborted (core dumped)

However, when running the same test on a local Minikube setup, it would set it to 100Mi, and the test would pass.

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-my-cluster-1804135696-zookeeper-0   Bound    pvc-eb1ba5df-ebd4-4399-b17c-46a200ca6178   100      RWO            standard       26s
data-my-cluster-1804135696-zookeeper-1   Bound    pvc-e0416ad0-a839-4984-b263-1c14b967a90f   100      RWO            standard       26s
data-my-cluster-1804135696-zookeeper-2   Bound    pvc-d841b6c0-2e56-4249-a25b-c14182d09be5   100      RWO            standard       26s

To resolve this issue, we had to add the required unit into strimzi-kafka-operator/systemtest/src/main/java/io/strimzi/systemtest/templates/crd/KafkaTemplates.java (lines 62 + 68).

   public static KafkaBuilder kafkaPersistent(String name, int kafkaReplicas, int zookeeperReplicas) {
        Kafka kafka = getKafkaFromYaml(Constants.PATH_TO_KAFKA_PERSISTENT_CONFIG);
        return defaultKafka(kafka, name, kafkaReplicas, zookeeperReplicas)
            .editSpec()
                .editKafka()
                    .withNewPersistentClaimStorage()
                        .withSize("100Mi")
                        .withDeleteClaim(true)
                    .endPersistentClaimStorage()
                .endKafka()
                .editZookeeper()
                    .withNewPersistentClaimStorage()
                        .withSize("100Mi")
                        .withDeleteClaim(true)
                    .endPersistentClaimStorage()
                .endZookeeper()
            .endSpec();
    }

To Reproduce Steps to reproduce the behavior: