Airflow .airflowignore not handling soft link properly.

Apache Airflow version

2.3.0 (latest released)

What happened

Soft link and folder under same root folder will be handled as the same relative path. Say i have dags folder which looks like this:

-dags:
  -- .airflowignore
  -- folder
  -- soft-links-to-folder -> folder

and .airflowignore:

folder/

both folder and soft-links-to-folder will be ignored.

What you think should happen instead

Only the folder should be ignored. This is the expected behavior in airflow 2.2.4, before i upgraded. The root cause is that both _RegexpIgnoreRule and _GlobIgnoreRule is calling relative_to method to get search path.

How to reproduce

check @tirkarthi comment for the test case.

Operating System

ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[tirkarthi]

Here is a sample test case of the report. I guess this is more with the fact resolve is called before match that will resolve symlink to the original folder here. As per report “soft-links-to-folder” will resolve to “folder” and get ignored

https://github.com/apache/airflow/blob/6cc41abf6912fd2705b9ef7cf368c888c43c8af8/airflow/utils/file.py#L68

Test case passes before changes in https://github.com/apache/airflow/pull/22051 . cc : @ianbuss

# tests/plugins/test_plugin_ignore.py
def test_symlink_not_ignored(self):
    shutil.rmtree(self.test_dir)
    self.test_dir = tempfile.mkdtemp(prefix="onotole")
    source = os.path.join(self.test_dir, "folder")
    target = os.path.join(self.test_dir, "symlink")
    py_file = os.path.join(source, "hello_world.py")
    ignore_file = os.path.join(self.test_dir, ".airflowignore")
    os.mkdir(source)
    os.symlink(source, target)

    with open(ignore_file, 'w') as f:
        f.write("folder")

    with open(py_file, 'w') as f:
        f.write("print('hello world')")


    ignore_list_file = ".airflowignore"
    found = []

    for path in find_path_from_directory(self.test_dir, ignore_list_file):
        found.append(path)

    assert os.path.join(self.test_dir, "symlink", "hello_world.py") in found

[ianbuss]

Have prepped a simple initial PR which should hopefully restore the original behaviour (and includes the test case provided by @tirkarthi - thanks!) but would be good to get some additional eyes on it. If we want to make larger changes to the symlink handling that should perhaps be a future PR with further thought? Depends on the timeline of 2.3.1 I think.