Deleting temporary directories early causes non-unique paths and other issues

With #10442, tmp_path paths are not necessarily unique anymore - in other words, this passed previously, but fails now:

some_global_state = set()


def test_some_very_long_test_name_first(tmp_path):
    print(tmp_path)
    some_global_state.add(tmp_path)


def test_some_very_long_test_name_second(tmp_path):
    print(tmp_path)
    assert tmp_path not in some_global_state

This is because those tests used to use unique temp directory names:

x.py::test_some_very_long_test_name_first /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
PASSED
x.py::test_some_very_long_test_name_second /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_1
FAILED

but now don’t do so anymore, instead, the second test reuses the path the first test did:

x.py::test_some_very_long_test_name_first /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
PASSED
x.py::test_some_very_long_test_name_second /tmp/pytest-of-florian/pytest-125/test_some_very_long_test_name_0
FAILED

In my case, this caused issues with using Qt’s QSqlDatabase, which has a list of connections as global state. Since with sqlite, a connection is based on its path, two tests used the same connection name.

I fixed it by closing the database connection properly after the test. This is definitely something I should have done before already.

However, this still violates a basic expectation on how tmp_path should behave, which is pointed out in the first sentence of the docs (emphasis mine):

You can use the tmp_path fixture which will provide a temporary directory unique to the test invocation

It’s of course questionable whether the “unique” refers to the directory contents, or the path to it - but as shown above, assuming the latter isn’t really something out of the ordinary, especially when passing that path to a library somewhere.

---

I’ve also run into another issue with the same commit, which I don’t even know how to describe. The gist of it that I had a Qt QFileSystemModel watching the temporary directory for changes - and when it was deleted after the test, Qt/PyQt somehow tried to call a callback lambda on a Python object which was already deleted, thus segfaulting…

This one smells like a PyQt bug, but again demonstrates how files and file paths are inherently attached to some side effects, and deleting them after the test has finished might be too early, depending on what code in the background is doing with it. This also makes me a bit worried about #10546, which I haven’t tried yet.

---

On a more personal note, I also wonder if it’s really the right decision to change the default behavior (policy failed), even more so in a non-major release. As demonstrated above, it can have very subtle changes on a testsuite.

But, perhaps even more important than that, this makes something I consider a great feature of pytest less discoverable. In various places (books, but also my trainings for example) the current behavior of pytest is taught to people, and by the time they discover those files aren’t actually kept around anymore, it might already be too late (if e.g. debugging a flaky test). It seems to me that this could lead to much more confusion and frustration compared to it being problematic when storing lots of big files there. The latter seems like a non-issue to me for most cases where I see tmp_path being used.

@yusuke-kadowaki @nicoddemus @RonnyPfannschmidt @hmaarrfk what do you think?

[nicoddemus]

Given all the information on the thread, I agree it is better to return to the old default. It has been working like that for many years and personally I also had no problems with it. The fact that this might cause headaches for many people once released is a real red flag: we can take the data point that just @The-Compiler already found two hard-to-track problems in his test suite alone. We should be conservative here and keep the old default and possible prevent a myriad of bug report / issues with users.

So IMHO we should do two things:

1. Change the default settings to mimic the old behavior.
2. Fix the issue that we broke the expectation of unique directories. While arguable if this is a feature or not, I would prefer to keep the previous behavior, otherwise people might get surprising test failures if the behavior changes between “keep” modes.

If nobody opposes it, we should create separate issues (linking to this one) and get volunteers to work on them, and close this.

[nicoddemus]

Plus why would one want to delete folders early instead of deleting at the end of the session? Both will have the same result in terms of the data usage when you see it after the test session finishes.

This is important to consider, because this automatically resolves the numbering issue. Is there a strong, compelling case for deleting folders as soon as the test finishes, as opposed to delay deletion to the end of the test session?