Failed to adopt pod: Scheduler cannot adopt running pods

Apache Airflow version

2.2.4

What happened

I am running Airflow 2.2.4 on Kubernetes, using the KubernetesExecutor. If I re-create the scheduler pod, it attempts to adopt running job pods but fails to do so:

[2022-05-10 11:35:23,707] {kubernetes_executor.py:714} INFO - Failed to adopt pod <REDACTED>. Reason: (422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ac152b63-74ef-48c2-b4eb-fe5fbc808a56', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '251e299d-3b5d-4c7a-a3d9-46f17a316c93', 'X-Kubernetes-Pf-Prioritylevel-Uid': '66005fda-3f10-4344-8170-8c819dbbf59f', 'Date': 'Tue, 10 May 2022 11:35:23 GMT', 'Transfer-Encoding': 'chunked'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"<REDACTED>\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations) <TRUNCATED>

The pods are then killed by the scheduler.

I tried to dig into the code as best I could, and I found that this might be caused by the KubernetesExecutor trying to update the pod’s metadata.labels here - but I could be wrong as I’m not very familiar with this part of Airflow.

What you think should happen instead

The scheduler should be able to adopt running pods instead of killing them.

How to reproduce

• Run Airflow with the KubernetesExecutor
• Start a long-running task
• Re-create the scheduler pod

Operating System

Debian GNU/Linux 10 (buster)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

This happens when the scheduler pod is re-created while a job pod is running.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[olivermeyer]

@potiuk happy to report that upgrading to 2.3.1 fixed the issue for us. Thank you!

[potiuk]

@potiuk happy to report that upgrading to 2.3.1 fixed the issue for us. Thank you!

Glad it helped 😃