Test Results directory in setup fails intermittently

This issue is not consistent, but sometimes the Test Results directories are not created or, at least, not created in time causing the setup to fail. This looks like an issue with race conditions, but right now we have

• pytest-xdist
• pytest-parallel

as the two plugins that allow you to run tests in parallel. This feature is considered “legacy” since it was one of the first features written. These folders were used to store log files and screenshots for each test, but logging now happens via the logging module so now these folders are only used for screenshots if the test fails.

So, we can either fix this issue, or we can create a new Feature Request suggesting a better, cleaner way to report logs and screenshots on failure.

project_root = '/home/runner/work/pyleniumio/pyleniumio'
request = <SubRequest 'test_run' for <Function test_element_parent_and_siblings>>

    @pytest.fixture(scope='session', autouse=True)
    def test_run(project_root, request) -> str:
        """ Creates the `/test_results` directory to store the results of the Test Run.
    
        Returns:
            The `/test_results` directory as a filepath (str).
        """
        session = request.node
        test_results_dir = f'{project_root}/test_results'
    
        if os.path.exists(test_results_dir):
            # delete /test_results from previous Test Run
            shutil.rmtree(test_results_dir, ignore_errors=True)
        if not os.path.exists(test_results_dir):
            # create /test_results for this Test Run
            make_dir(test_results_dir)
    
        for test in session.items:
            # make the test_result directory for each test
>           make_dir(f'{test_results_dir}/{test.name}')

conftest.py:111: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

filepath = '/home/runner/work/pyleniumio/pyleniumio/test_results/test_element_with_no_siblings'

    def make_dir(filepath) -> bool:
        """ Make a directory.
    
        Returns:
            True if successful, False if not.
        """
        try:
>           os.mkdir(filepath)
E           FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/pyleniumio/pyleniumio/test_results/test_element_with_no_siblings'

[ElSnoMan]

@chingc I believe I’ve fixed this issue in v1.12.5, but I’d love for you to try it out! I always appreciate your feedback 😄

[chingc]

@ElSnoMan Nevermind, I just hit this today. 😦

I’m running using pytest -n 5 so I think what’s happening is the 5 workers clobbering each other.

I think the race condition is happening between the time Path is checking to see if the file exists and when it attempts to create it. I think adding a try/except might fix this problem for good. For example:

for test in session.items:
    try:
        # make the test_result directory for each test
        Path(f'{test_results_dir}/{test.name}').mkdir(parents=True, exist_ok=True)
    except FileExistsError:
        # avoid race condition between checking file existence and creating the file when using pytest with multiple workers
        pass

Below is the error I got.

self = PosixPath('/home/ubuntu/bentobox/test_results/test_patch_is_active')
mode = 511, parents = True, exist_ok = True

    def mkdir(self, mode=0o777, parents=False, exist_ok=False):
        """
        Create a new directory at this given path.
        """
        if self._closed:
            self._raise_closed()
        try:
>           self._accessor.mkdir(self, mode)
E           FileExistsError: [Errno 17] File exists: '/home/ubuntu/bentobox/test_results/test_patch_is_active'

/usr/lib/python3.8/pathlib.py:1284: FileExistsError
_ ERROR at setup of GoogleInventoryAPIServiceTestCase.test_incorrect_update_response_will_log_warning _
[gw1] linux -- Python 3.8.5 /home/ubuntu/venv/bin/python3

project_root = '/home/ubuntu/bentobox'
request = <SubRequest 'test_run' for <TestCaseFunction test_update_when_valid>>

    @pytest.fixture(scope='session', autouse=True)
    def test_run(project_root, request) -> str:
        """Creates the `/test_results` directory to store the results of the Test Run.
    
        Returns:
            The `/test_results` directory as a filepath (str).
        """
        session = request.node
        test_results_dir = f'{project_root}/test_results'
    
        if os.path.exists(test_results_dir):
            # delete /test_results from previous Test Run
            shutil.rmtree(test_results_dir, ignore_errors=True)
    
        Path(test_results_dir).mkdir(parents=True, exist_ok=True)
    
        for test in session.items:
            # make the test_result directory for each test
>           Path(f'{test_results_dir}/{test.name}').mkdir(parents=True, exist_ok=True)

conftest.py:99: