dvc: .dvc/lock and sqlite lock on NFS and CIFS

As it turns out, regular locks can’t be relied on when dvc is running on NFS or CIFS. See https://github.com/iterative/dvc/issues/1823 and https://discordapp.com/channels/485586884165107732/485596304961962003/570270243964846081 .

There are workarounds such as adding proper mnt options or moving dvc project outside with an external cache directory at NFS/CIFS mount, but it would be great to mitigate such issues in general. Here are a few ways to go about it:

One way to go about it is to use git-like locks (which, if I recall correctly, are using symlinks as an atomic way lock/unlock a file). In that case, we would have to use an unlocked sqlite db relying on our .dvc/lock (or maybe introduce a separate special lock specifically for a db).

[Suor]

The questions need to be answered:

• can we use symlink locks like git does? Are there any ready to use python implementation? Do NFS/CIFS/LustreFS support atomic symlinks?
• are there some key-value file backed db which we can switch to from sqlite which works on NFS/CIFS/…?
• where do we place sqlite db, if we don’t place it within .dvc?

[Suor]

Some takeouts from discussion with @efiop:

• if we have good universal locks then this is rather straightforward: use sqlite nolock option, use those locks around sqlite and in place of any current locks.
• git uses open(O_EXCL | O_CREAT | ...) locks, which don’t work on pre v3 NFS.
• accesibility of symlinks rely on mount options.
• flufl locks linked by @efiop above.