[bug] Repo advisory locks fail to release on process termination

I am occasionally seeing Conan get stuck trying to obtain the advisory locks on the repository or packages within the repository. I have dug through the code and I have a hard time understanding how this issue can possibly occur. Perhaps someone else can figure out what is happening, or maybe know’s the next place to look? Here’s what I know so far:

I have seen issue #2071, but I do not believe it to be fixed on all platforms, and I do not understand how PR #2996 could have “fixed” the issue:

• Conan, as far as I can tell, uses the “correct” system interprocess locking mechanism (not the naive method using open and O_EXCL).
• On Unix systems, this will use fcntl() to implement shared/exclusive locking with the support of the operating system.
• On Windows, this will use the msvcrt.locking Python function, which (AFAICT) is implemented in terms of Win32 LockFileEx.
• Regardless of platform, the operating system will guarantee that it will automatically release any open locks in case that the owning processes are terminated, regardless of termination method.
• For this reason, even if a lock owner is hard-killed with SIGKILL or ::TerminateProcess(), the advisory locks will be released by the operating system. Thus I cannot see how handling SIGTERM could have had any effect on the locking behavior.
• What’s more, I don’t see how this could possibly be an issue at all. By my understanding of the Win32 and POSIX documentation, there should be no way that the hold on the locks can possibly outlive a Conan process!
• I’m currently debugging on Windows, although I have seen (but cannot reproduce) the issue on Linux (I wouldn’t worry about Linux ATM).
• I opened ProcExp to try and see what process is holding open a handle to the package lock (foo.count.lock), and… nothing. No one has the file open! AFAIK, if no one has the file open, the operating system will release the lock. What gives?

I’ve opened the code to the MSVCRT, locking.cpp, upon which msvcrt.locking is implemented. It all looks like completely valid usage of ::LockFileEx, and I cannot see how the locks can outlive the process.

Environment Details (include every applicable attribute)

• Operating System+version: Windows Server 1903 Core
• Conan version: 1.20.2
• Python version: Python 3.8.0

Steps to reproduce (Include if Applicable)

1. Begin a conan operation to install/build/export a package into the local repository.
2. While an advisory lock is held, non-gracefully terminate the Python process. On Windows, there are no ‘signals’ quite like POSIX, although there is limited emulation of SIGINT. I cannot be certain what termination mechanism is in use, as this termination is going through “Stop build” in a Bamboo dashboard, of which the documented behavior on Windows is somewhat confusing. However, the termination mechanism should not matter (See above).
3. Attempting to re-run the process will cause Conan to deadlock when trying to re-obtain access to the same advisory locks.

[vector-of-bool]

The fasteners project recently merged some changes that use proper OS-supported advisory locking. I don’t know if they are released yet, but using the newer version of fasteners will give access to its more reliable locking.

[vector-of-bool]

I’ve very recently dealt with writing shared locking on both Windows and Linux, and the code itself is both simple and reliable. fasteners’ reception to PRs seems terribly slow, and I’m not hopeful in getting a change in. No existing library I can find implements shared locking (correctly, at least). I may intern some of the locking code and write some thorough tests on it. Stay tuned.