InterProcessLock files left behind

When trying to use the InterProcessLock class on OS X it appears that when it releases the lock, it doesn’t cleanup after itself. It would be great if it could do this to ensure that we don’t leave a bunch of left over files laying around.

[tgamblin]

I was able to find a decent solution for this for my use case by using byte range locking with Python’s lockf (which is confusingly fcntl underneath).

In particular, I have a bunch of install prefixes for different packages in a package manager, and each needs to be write-locked while an install is happening and read-locked while a dependency’s install is happening. Lots of installs can happen in parallel.

I use a single lock file, and for each directory, I lock a single byte in the lock file identified by the 63-bit prefix of the SHA-1 of the directory name.

Yes, that is complicated, but it has some advantages:

1. Uses fcntl locks, which seems to be reasonably supported across distributed filesystems I care about (NFS v2.6+, Lustre, GPFS).
2. Only requires one lock file, whose actual size is zero since you can lock past the end of the file.
3. No need to clean up the one lock file (cleaning up fcntl locks gives you race conditions anyway).
4. Interprocess reader-writer locks.

Disadvantages:

1. This approach is not “perfect”, in the sense that it can result in false positives – if there is a hash collision you may have to wait for an install to finish when you do not need to wait. But the likelihood of a collision with two locking processes and 64-bit offsets (most machines) is 1e-19, and the likelihood of a collision doesn’t reach 1% until you have ~430 million processes (at which point you probably have other problems).
2. I think if you adapted this to also support msvcrt on Windows, you’d have to make all the locks exclusive, or maybe unix clients could use the read locks and they would appear to be exclusive locks to a windows client. I haven’t looked into how different NFS implementations handle this.

At least for my use case, the advantages outweigh the disadvantages, and I couldn’t come up with a better solution that didn’t require me to deploy a distributed lock server or some other coordination service.

Anyway, an interprocess reader-writer lock with byte range support is implemented here:

• https://github.com/LLNL/spack/blob/develop/lib/spack/llnl/util/lock.py
• https://github.com/LLNL/spack/blob/develop/lib/spack/spack/test/lock.py

There is some relevant discussion here. The SHA stuff is outside the lock class, so it probably doesn’t need to go in fasteners, unless you like the idea and want some kind of lock_by_name_sha1() method.

Just wanted to throw a potential solution in the thread. Let me know if you’d have a use for this in fasteners.

[vstinner]

Would it make sense to use a file descriptor rather than a filename with O_TMPFILE, to ensure that the file is destroyed as soon at the file is closed in all processes (or when processes are killed)? O_TMPFILE requires Linux 3.11 and newer, and is not supported by all filesystems, but more and more filesystems support it.

See how tempfile of Python stdlib uses O_TMPFILE:

• https://github.com/python/cpython/blob/70f61b933f5f4bbb28d2c1a2ebf3a555513e67b0/Lib/tempfile.py#L565
• https://github.com/python/cpython/commit/d967fc9da2c5c094865514bd213dd702698e4dad
• https://bugs.python.org/issue21515

I just tested on Fedora 29: btrfs, ext4 and tmpfs support O_TMPFILE. (I only tested these filesystems.)

$ cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)

$ uname -r
4.18.16-300.fc29.x86_64

$ python3
Python 3.7.1 (default, Nov  5 2018, 14:07:04) 
>>> import os

>>> fd=os.open(".", os.O_WRONLY | os.O_TMPFILE) # brtfs
>>> os.close(fd)

>>> fd=os.open("/tmp", os.O_WRONLY | os.O_TMPFILE) # tmpfs
>>> os.close(fd)

>>> fd=os.open("/boot/test", os.O_WRONLY | os.O_TMPFILE) # ext4
>>> os.close(fd)

Problem: who create the FD? How to pass the FD to other processes? UNIX socket with sendmsg()? …