Can we add configable debug settings for delay pod delete when there is a `Error` state of pods ?

Description

Add configable debug settings for delay pod delete when there is a Error state of pods.

Use case / motivation

In apache/airflow:1.10.10 image.

I’m deploy a airflow in k8s, want to use Kubernetes Executor for task excute. If the pod got Error state, airflow scheduler would delete pod immediately. So we can not see what happend, pod is deleted in some seconds.

When I add time.sleep() in kubernetes_executor.py:896 , like this:

    def _change_state(self, key, state, pod_id, namespace):
        if state != State.RUNNING:
            if self.kube_config.delete_worker_pods:
                for x in range(120):
                    self.log.info(str(x) + ": sleep 1s for...")
                    time.sleep(1)
                self.kube_scheduler.delete_pod(pod_id, namespace)
                self.log.info('Deleted pod: %s in namespace %s', str(key), str(namespace))
            try:
                self.running.pop(key)
            except KeyError:
                self.log.debug('Could not find key: %s', str(key))
        self.event_buffer[key] = state

When trigger execute manully, I can see pod got Error state soon.

➜  ~ kubectl get po
NAME                                                         READY   STATUS    RESTARTS   AGE
airflow-564c84ff46-tn5mg                                     2/2     Running   0          67s
examplebashoperatorrunme0-76fd68aa96d64e8c93c7c87904f3312a   0/1     Error     0          24s

Watch pod’s log:

➜  ~ kubectl logs -f examplebashoperatorrunme0-76fd68aa96d64e8c93c7c87904f3312a
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 23, in <module>
    import argcomplete
ModuleNotFoundError: No module named 'argcomplete'

It’s a error in container. It’s easy to debug now.

[hcbraun]

AIRFLOW__KUBERNETES__RUN_AS_USER: “50000”

[hcbraun]

Hi gwind, how did you solve the container error? ModuleNotFoundError: No module named 'argcomplete' I have the same issue in pods with the Kubernetes executor and the example DAGs

There is an option to keep / not delete worker pods: AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: “false”