pytest hangs if it tries to rewrite modules without having access rights on Windows 10

Basically, it seems that if:

• python is installed in C:\Program files (in a situation that you also have to run pip install from an elevated command prompt in order to install packages)
• pytest runs (as a normal user) for the first time and tries to rewrite assertions, without having elevation

Then pytest just hangs. CTRL+C shows the following traceback:

Traceback (most recent call last):
  File "c:\program files\python36\lib\tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python36\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 55, in main
    config = _prepareconfig(args, plugins)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 200, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "c:\program files\python36\lib\site-packages\pluggy\hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 203, in _multicall
    gen.send(outcome)
  File "c:\program files\python36\lib\site-packages\_pytest\helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 661, in pytest_cmdline_parse
    self.parse(args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 869, in parse
    self._preparse(args, addopts=addopts)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 815, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 292, in load_setuptools_entrypoints
    plugin = ep.load()
  File "c:\program files\python36\lib\site-packages\importlib_metadata\__init__.py", line 90, in load
    module = import_module(match.group('module'))
  File "c:\program files\python36\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 144, in exec_module
    _write_pyc(state, co, source_stat, pyc)
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 267, in _write_pyc
    with atomicwrites.atomic_write(pyc, mode="wb", overwrite=True) as fp:
  File "c:\program files\python36\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 156, in _open
    with get_fileobject(**self._open_kwargs) as f:
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 172, in get_fileobject
    descriptor, name = tempfile.mkstemp(dir=dir)
  File "c:\program files\python36\lib\tempfile.py", line 344, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "c:\program files\python36\lib\tempfile.py", line 268, in _mkstemp_inner
    if (_os.name == 'nt' and _os.path.isdir(dir) and
KeyboardInterrupt

I would expect it not to hang, but for it to terminate with a Permission denied message instead.

Using Windows 10, python 3.6.8 and pytest 5.0.0

[nicoddemus]

Hi @Sup3rGeo!

atomicwrites tries to create the temporary directory 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e' using tempfile.mkstemp, which in turn calls _mkstemp_inner:

def _mkstemp_inner(dir, pre, suf, flags, output_type):
    """Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""

    names = _get_candidate_names()
    if output_type is bytes:
        names = map(_os.fsencode, names)

    for seq in range(TMP_MAX):
        name = next(names)
        file = _os.path.join(dir, pre + name + suf)
        try:
            fd = _os.open(file, flags, 0o600)
        except FileExistsError:
            continue    # try again
        except PermissionError:
            # This exception is thrown when a directory with the chosen name
            # already exists on windows.
            if (_os.name == 'nt' and _os.path.isdir(dir) and
                _os.access(dir, _os.W_OK)):
                continue
            else:
                raise
        return (fd, _os.path.abspath(file))

    raise FileExistsError(_errno.EEXIST,
                          "No usable temporary file name found")

The traceback shows it is hanging on the PermissionError handling, which does call _os.access(dir, _os.W_OK) to check if it should keep retrying to create the temporary directory, otherwise it fails. It seems for some reason it is keeping looping there, instead of _os.access(dir, _os.W_OK) returning False and we getting the expected PermissionError.

Strange, unfortunately this is hard to debug. 😕

[marcel-kanter]

Stumbled across this too… There are two issues:

1. The assumption with the permission error
2. the pyc-path may be read-only for normal users (this may be on linux too?)
3. get_fileobject of atomicwrites gets called without a dir from assertion rewrite

os.fspath(pyc) may be read-only (on linux too?) and this arguments does not go to get_fileobject (this is called with dir empty) and then is replaces it (which is dangerous, since this may be read-only too)

    if dir is None:
        dir = os.path.normpath(os.path.dirname(self._path))

printing dir after these two lines in atomicwrites/init.py

gives me C:\Program Files\Python38\lib\site-packages\hypothesis_pycache_

with is write protected.

File “C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py”, line 165, in exec_module _write_pyc(state, co, source_stat, pyc) File “C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py”, line 308, in write_pyc with atomic_write(os.fspath(pyc), mode=“wb”, overwrite=True) as fp: File “C:\Program Files\Python38\lib\contextlib.py”, line 113, in enter return next(self.gen) File "C:\Program Files\Python38\lib\site-packages\atomicwrites_init.py", line 166, in _open with get_fileobject(**self.open_kwargs) as f: File "C:\Program Files\Python38\lib\site-packages\atomicwrites_init.py", line 190, in get_fileobject

Calling pytest with --assert=plain is a work-around.