KubernetesPodOperator with pod_template_file = No Metadata & Wrong Pod Name

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) 1.15.15

What happened:

If you use the KubernetesPodOperator with LocalExecutor and you use a pod_template_file, the pod created doesn’t have metadata like :

• dag_id
• task_id
• …

I want to have a privileged_escalation=True pod, launched by a KubernetesPodOperator but without the KubernetesExecutor. Is it possible ?

What you expected to happen:

Have the pod launched with privileged escalation & metadata & correct pod-name override.

How to reproduce it:

• have a pod template file :

privileged_runner.yaml :

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  containers:
  - name: base
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true

• have a DAG file with KubernetesOperator in it :

my-dag.py :

##=========================================================================================##
##                                      CONFIGURATION 

from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.kubernetes.secret import Secret
from kubernetes.client import models as k8s
from airflow.models import Variable
from datetime import datetime, timedelta
from airflow import DAG

env = Variable.get("process_env")
namespace = Variable.get("namespace")

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

##==============================##

## Définition du DAG
dag = DAG(
    'transfert-files-to-nexus',
    start_date=datetime.utcnow(),
    schedule_interval="0 2 * * *",
    default_args=default_args,
    max_active_runs=1
)
##=========================================================================================##

## Définition des tâches
start = DummyOperator(task_id='start', dag=dag)
end = DummyOperator(task_id='end', dag=dag)

transfertfile = KubernetesPodOperator(namespace=namespace,
                             task_id="transfertfile",
                             name="transfertfile",
                             image="registrygitlab.fr/docker-images/python-runner:1.8.22",
                             image_pull_secrets="registrygitlab-curie",
                             pod_template_file="/opt/bitnami/airflow/dags/git-airflow-dags/privileged_runner.yaml",
                             is_delete_operator_pod=False,
                             get_logs=True,
                             dag=dag)
                             
## Enchainement des tâches
start >> transfertfile >> end

Anything else we need to know: I know that we have to use the KubernetesExecutor in order to have the metadata, but even if you use the KubernetesExecutor, the fact that you have to use the pod_template_file for the KubernetesPodOperator makes no change, because in either LocalExecutor / KubernetesExecutoryou will endup with no pod name override correct & metadata.

[armandleopold]

@kaxil Haven’t got the time to do so, a little busy at the moment. Not sure if i can try things for now.

[armandleopold]

Found a wordaround using full_pod_spec

full_pod_spec=k8s.V1Pod(
metadata=k8s.V1ObjectMeta(
name="hardcodedname"),
spec=k8s.V1PodSpec(
containers=[k8s.V1Container(
name="base",
security_context=k8s.V1SecurityContext(
allow_privilege_escalation=True,privileged=True))])),