Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Strimzi kafka operator pod getting evicted EmptyDir volume "strimzi-tmp" exceeds the limit

See original GitHub issue

Describe the bug Strimzi kafka operator pod getting evicted Usage of EmptyDir volume "strimzi-tmp" exceeds the limit "1Mi".

To Reproduce Steps to reproduce the behavior:

Create kafka clusters that can’t be scheduled ( causing alot of java error in operator logs )
Wait

Expected behavior Allow customisation of strimzi-tmp limit

Environment (please complete the following information):

Strimzi version: 0.25.0
Installation method: Helm chart
Kubernetes cluster: Kubernetes 1.19.1

Issue Analytics

State:
Created 2 years ago
Comments:30 (16 by maintainers)

Top GitHub Comments

4reactions

kyguycommented, Nov 3, 2021

Can you check if CC properly cleans the certificate stores?

@scholzj Argh this is my fault, I added the call to create the CC tmp file here [1] but I never added any calls to delete it. I’ll submit a PR to fix this!

[1] https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/resource/cruisecontrol/CruiseControlApiImpl.java#L50

3reactions

Lykos84commented, Jan 13, 2022

Ok, after some investigation internally, and a great tutorial on reading the documentation from a friend here at work, documentation here we realized that the tmpDirSizeLimit should be applied at the kafka spec level as a variable under template. For example:

spec:
  kafka:
    template:
      pod:
        tmpDirSizeLimit: 100Mi

This solved our issue. I have no idea if that helm chart value I mentioned before is supposed to apply to all pods that use the pod template, but now that I understand how this functions at the pod template level, it’s definitely applying as I would expect when deploying a kafka resource.

Thanks for your help in sorting all this out, @scholzj.

Top Results From Across the Web

In AMQ Streams, the kafka, zookeeper, entity operator pods ...

Issue. AMQ Streams pods are Evicted, and under "Events" it shows: Usage of EmptyDir volume "strimzi-tmp" exceeds the limit "5Mi".

Configuring Strimzi (In Development)

Ephemeral storage uses emptyDir volumes to store data. An emptyDir volume is created when a pod is assigned to a node. You can...

Using Strimzi

Ephemeral storage uses emptyDir volumes to store data. ... not exceed access to Kafka brokers based on a byte threshold or a time...

https://strimzi.io/install/latest?namespace=kafka

A Pod eviction is allowed when the `maxUnavailable` number of pods or fewer are ... strimzi-cluster-operator volumes: - name: strimzi-tmp emptyDir: medium: ...

Deploying and Upgrading (0.32.0) - Strimzi

For example, kubectl get kafkatopic my-topic or kubectl get kafkatopics . ... Using an emptyDir volume means that its content is strictly related...