Noobaa endpoint pod Terminated before new one comes in Running state

Environment info

• NooBaa Version: ODF 4.9.0 - 236.ci
• Platform: OCP 4.9

Actual behavior

1. The endpoint pods goes into terminating state

Expected behavior

1. The old endpoint should come to Terminating after new ones in running state

Steps to reproduce

1. Changed noobaa log level

[root@ocp-akshat-1-inf akshat]# oc get pod
NAME                                               READY   STATUS    RESTARTS   AGE
noobaa-core-0                                      1/1     Running   0          6m45s
noobaa-db-pg-0                                     1/1     Running   0          6m45s
noobaa-default-backing-store-noobaa-pod-41376b30   1/1     Running   0          4m46s
noobaa-endpoint-6b98668666-wkjvq                   1/1     Running   0          45s
noobaa-operator-67c57b5464-wqcww                   1/1     Running   0          4h30m
ocs-metrics-exporter-6fc4cbfcb6-qc7vx              1/1     Running   0          4h30m
ocs-operator-75fc898954-9cqt5                      1/1     Running   0          4h30m
odf-console-7dc4779787-qv44z                       1/1     Running   0          4h30m
odf-operator-controller-manager-548c77868b-nb87q   2/2     Running   0          4h30m
rook-ceph-operator-c796c8c98-nqd8f                 1/1     Running   0          4h30m
[root@ocp-akshat-1-inf akshat]# kubectl edit configmap/noobaa-config
configmap/noobaa-config edited
[root@ocp-akshat-1-inf akshat]# 
[root@ocp-akshat-1-inf akshat]# 
[root@ocp-akshat-1-inf akshat]# oc get pod
NAME                                               READY   STATUS              RESTARTS   AGE
noobaa-core-0                                      1/1     Terminating         0          7m12s
noobaa-db-pg-0                                     1/1     Running             0          7m12s
noobaa-default-backing-store-noobaa-pod-41376b30   1/1     Running             0          5m13s
noobaa-endpoint-6b98668666-px86w                   0/1     ContainerCreating   0          2s    ----->>> This one       
noobaa-endpoint-6b98668666-wkjvq                   1/1     Terminating         0          72s
noobaa-operator-67c57b5464-wqcww                   1/1     Running             0          4h30m
ocs-metrics-exporter-6fc4cbfcb6-qc7vx              1/1     Running             0          4h30m
ocs-operator-75fc898954-9cqt5                      1/1     Running             0          4h30m
odf-console-7dc4779787-qv44z                       1/1     Running             0          4h31m
odf-operator-controller-manager-548c77868b-nb87q   2/2     Running             0          4h31m
rook-ceph-operator-c796c8c98-nqd8f                 1/1     Running             0          4h30m
[root@ocp-akshat-1-inf akshat]# 

More information - Screenshots / Logs / Other output

[dannyzaken]

@guymguym @akmithal so what I think happens here is that we explicitly delete the endpoint pods when the config map is changed so the pods will use the new envs. kuberentes does not automatically restart the pods on config maps changes, so in this case the strategy is not relevant

see here (line 1486):

https://github.com/noobaa/noobaa-operator/blob/207b8f5dbd0d1ee73363d10541a626a6a5eeb661/pkg/system/phase4_configuring.go#L1470-L1488

[akmithal]

One more observation: This behaviour is observed only with step “kubectl edit configmap/noobaa-config”.

When I edit deployment to change CPU/Mem resources, the new endpoint pod first comes into Running state and then the old one goes into Terminating( this is expected behaviour)