tempfile.TemporaryDirectory does not get deleted after task failure

Discussed in https://github.com/apache/airflow/discussions/22403

<div type='discussions-op-text'>

^{Originally posted by m1racoli March 18, 2022}

Apache Airflow version

2.2.4 (latest released)

What happened

When creating a temporary directory with tempfile.TemporaryDirectory() and then failing a task, the corresponding directory does not get deleted.

This happens in Airflow on Astronomer as well as locally in for astro dev setups for LocalExecutor and CeleryExecutor.

What you think should happen instead

As in normal Python environments, the directory should get cleaned up, even in the case of a raised exception.

How to reproduce

Running this DAG will leave a temporary directory in the corresponding location:

import os
import tempfile

from airflow.decorators import dag, task
from airflow.utils.dates import days_ago


class MyException(Exception):
    pass


@task
def run():
    tmpdir = tempfile.TemporaryDirectory()
    print(f"directory {tmpdir.name} created")
    assert os.path.exists(tmpdir.name)

    raise MyException("error!")


@dag(start_date=days_ago(1))
def tempfile_test():
    run()


_ = tempfile_test()

Operating System

Debian (Astronomer Airflow Docker image)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==1!3.0.0
apache-airflow-providers-cncf-kubernetes==1!3.0.2
apache-airflow-providers-elasticsearch==1!2.2.0
apache-airflow-providers-ftp==1!2.0.1
apache-airflow-providers-google==1!6.4.0
apache-airflow-providers-http==1!2.0.3
apache-airflow-providers-imap==1!2.2.0
apache-airflow-providers-microsoft-azure==1!3.6.0
apache-airflow-providers-mysql==1!2.2.0
apache-airflow-providers-postgres==1!3.0.0
apache-airflow-providers-redis==1!2.0.1
apache-airflow-providers-slack==1!4.2.0
apache-airflow-providers-sqlite==1!2.1.0
apache-airflow-providers-ssh==1!2.4.0

Deployment

Astronomer

Deployment details

GKE, vanilla astro dev, LocalExecutor and CeleryExecutor.

Anything else

Always

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

</div>

[m1racoli]

Awesome! Great work @potiuk!

[potiuk]

Really nice one - fix in #22475 😄 - tested it with:

import os
from time import sleep
import tempfile


def test():
    tmpdir = tempfile.TemporaryDirectory()
    print(f"directory {tmpdir.name} created")
    assert os.path.exists(tmpdir.name)
    raise Exception("exiting")


pid = os.fork()
if pid:
    sleep(2)
else:

    try:
        test()
    except BaseException:
        pass
    finally:
        pass
    os._exit(0)