Allow deleting existing spark application before creating new one via SparkKubernetesOperator in Kubernetes

airflow version: v2.0.2

Description calling SparkKubernetesOperator within DAG should delete spark application if such already exists before submitting a new one.

Use case / motivation

t1 = SparkKubernetesOperator(
    task_id='spark_pi_submit',
    namespace="dummy",
    application_file="spark.yaml",
    kubernetes_conn_id="kubernetes",
    do_xcom_push=True,
    dag=dag,
)

After first successful run, next runs fail to submit spark application

airflow.exceptions.AirflowException: Exception when calling -> create_custom_object: (409) Reason: Conflict

{“kind”:“Status”,“apiVersion”:“v1”,“metadata”:{},“status”:“Failure”,“message”:“sparkapplications.sparkoperator.k8s.io “xxx” already exists”,“reason”:“AlreadyExists”,“details”:{“name”:“xxx”,“group”:“sparkoperator.k8s.io”,“kind”:“sparkapplications”},“code”:409}

Expected Result

Delete existing spark application if such exists before submitting new one.

[jherrmannNetfonds]

Hi, I modified the SparkKubernetesSensor so it deletes the spark application if it finished successfully. For retry I have {{ task_instance.try_number }} for the SparkApplication name so that I still can have a look at the logs.

My proposal is to add an option to the Sensor to delete the spark application via a flag ALWAYS, ON_FAILURE,ON_SUCCESS. What do you guys think?

[avinovarov]

Bumping into same issue, would be happy to have it fixed, thanks!