[bug] access denied while renaming unzipped folders

This issue is the same than https://github.com/conan-io/conan/issues/5205

Environment Details (include every applicable attribute)

• Operating System+version: Windows 10
• Conan version: 1.22.2
• Python version: 3.5.4

• Windows indexing is disabled for .conan folder

Steps to reproduce (Include if Applicable)

• run conan create or conan install such a way it has to build from source.
• use recipe and version where source code is a big package (in size or number of files), for me it happens all the time with physx and cspice (waiting merge in conan-center-index), sometimes with qt or boost, and with different computers (but always on Windows 10).

Logs (Executed commands with output) (Include/Attach if Applicable)

ERROR: physx/4.1.1: Error in source() method, line 80
        os.rename(extracted_dir, self._source_subfolder)
        PermissionError: [WinError 5] Access denied: 'PhysX-ae80dede0546d652040ae6260a810e53e20a06fa' -> 'source_subfolder'

Adding a try / except + sleep few seconds is a workaround, since this permission denied seems to be ephemeral, or a loop with few retries.

It would be nice if conan could provide a tools.rename in order to handle these kinds of issues on Windows.

[jokram]

In another context I’ve observed the permission denied error and there I’ve tried to isolate the problem. With the following python script I was able to produce the error nearly 100%:

import errno
import os
import platform
import shutil
import stat
import time


def isWindows():
  """ Returns True when running with the native port of Python for Windows,
  False when running on any other platform (including the Cygwin port of
  Python).
  """
  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
  return platform.system() == "Windows"


def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      else:
        raise
  else:
    os.rename(src, dst)


def rmtree(path, ignore_errors=False):
  """shutil.rmtree(path) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  onerror = None
  if isWindows():
    path = path
    onerror = handle_rmtree_error
  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)


def handle_rmtree_error(function, path, excinfo):
  # Allow deleting read-only files
  os.chmod(path, stat.S_IWRITE)
  function(path)


def rename_test():
  if os.path.isdir("old_dir"):
    rmtree("old_dir")
  os.mkdir("old_dir")
  for i in range(100):
    file = "old_dir/test_" + str(i).zfill(3) + ".txt"
    #print("Generate " + file)
    with open(file,"w") as fp:
      for n in range(100):
        fp.write(str(i).zfill(3) + "." + str(n).zfill(3) + ": 01234567890 01234567890 0123456789\n")
  print("Rename old_dir => new_dir")
  rename("old_dir", "new_dir")
  rmtree("new_dir")


# Problem occured for: D:/ma/pkg/baseswp\\.git.tmp' -> 'D:/ma/pkg/baseswp\\.git
for i in range(100):
  print("Rename test " + str(i))
  rename_test()

• I’ve run the tests also in a directory where no Windows indexing service is looking into => error still occurs
• I’ve checked the Virus Scanner logs, but could not find an entry

=> So my conclusion so far is that maybe the file operations (before the call to os.rename()) return already before the file handle is released (async.).

As a workaround I’ve implemented a retry loop with waiting 500 ms between retries:

def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      elif e.errno == errno.EACCES:
        # Workaround for the sporadic problem on Windows:
        # PermissionError: [WinError 5] Access denied
        # Retry 3 times to rename with sleeping 500 ms in between
        retry = 3
        while retry:
          #print("*** Retry: " + str(4 - retry))
          time.sleep(0.5)
          try:
            os.rename(src, dst)
            retry = 0
          except OSError as e:
            if retry and e.errno == errno.EACCES:
              retry = retry - 1
            else:
              raise
      else:
        raise
  else:
    os.rename(src, dst)

[SpaceIm]

I’ve checked again a conan create with physx and antivirus disabled (on my personnal computer, can’t do that on corporate desktop), I don’t have permission denied error in this case.