Locking in GVFS

I’ve been working for a while to figure out the best way to support Git for game development. Right now the vast majority of game development studios use Perforce, which is closed-source, poorly implemented, and extremely expensive (causing many studios to use old builds from 2012 rather than upgrade).

The requirements for game development essentially boil down to two concerns:

• Suitable performance for large binary file revisioning Most of the assets that are committed on a daily basis are binary files. Images, Music, Animations, Rigs, Models can all easily take up 20-100GB at the head revision for a single game. There’s really no way to get around the size of assets here, especially for formats like audio.
• File-Locking The above formats are all fundamentally un-mergeable data types. ie. There’s fundamentally no way you can project a .wav file into a text file and merge it by hand, and there’s no way to generate the audio at build time – it’s a static file type. Working under this assumption, the only suitable option for avoiding conflicts (and potentially redoing days of work) is to prevent concurrent file modifications before they happen: File Locking.

GVFS (and GitLFS for the most part) comfortably cover the performance aspect here, so the main missing piece is the file locking behavior. As I see it, a good file-locking solution has a couple of requirements:

1. It must integrate well with existing software that may not expect files to be locked. Most programs and editors have no concept of file-locking. One main issue with Perforce is that it requires programs to add custom support for Locking/Unlocking a file on write. This leads to shoddily implemented source control plugins, and strictly limits the set of tools that can be used with file locking.
2. It must be total, or as close to total as possible. The system should strive to prevent as many concurrent modification issues as possible. For example, if two users simultaneously add file repo/fileA.bin, this should be prevented, or warned.
3. It must support normal git workflow. Locking should be implemented in a git-friendly methodology. Ideally, a code developer that doesn’t touch binaries files wouldn’t even notice the lock system’s presence. Locking should not assume a “master” branch and generally should work at the commit object level. (ie. a file can only be locked if you have checked out a descendant commit of the most recent modification). Locking should play nicely with normal git actions like reset, merge, branching, etc.
4. It must be straightforward enough for non-technical workers to manage. It must be simple enough to provide a clean UI to artists, or be invisible enough that it doesn’t take much understanding of git. Ideally, it is transparent enough to simply use SourceTree or other git visual clients.

Locking at the File System Level

I’ve thought a lot about the level at which a locking system should exist. Right now most locking systems operate at the user-level, marking files read-only, and requiring editors to manually unlock files before writing (see P4VS). They fail points 1 and 2. In order to satisfy points 1 and 2 above, I believe locking needs to act at the File System level.

Fundamentally, locking is a constraint on the file system, and not a separate tooling on top of it.

I think an ideal system for locking is where when opening a file, the file system would automatically lock the path for writes, similar to the lazy loading of files on a network drive. Given no lock contention, a program would operate on files as normal, without needing knowledge of the underlying locking system. Given contention, programs would interact via the normal file system errors that already exist for writing to non-writable files. Further, if we want to support the case in point 2, the file system is really the only place this can happen. It would need to recognize when new files are created and lock on the specific file path.

GVFS

Presumably, this could be implemented as an additional filter after GVFS (although I’m not particularly familiar with file system drivers). GVFS is already integrating with both the file-system and git, though, which makes it a fairly comfortable spot to place this functionality. Additionally, I believe file locking may require a couple of git modifications, which would need to be merged with the modifications made for GVFS.

I’ve only skimmed the GVFS source, so my understanding of the changes needed is limited, however I think roughly it needs a couple of things:

• A shared ledger of acquired locks. An append-only list of the path locks taken and released and which commits they are taken at.
• Hooks at the file system level to:
• Write to this ledger on file modification. (Lock a file on write, create, or delete)
• Reject writes, create, delete operations on lockable files, unless the HEAD commit is a descendant of the most recent commit on the ledger for that file.

I’d love to hear more thoughts on locking in general. My goal is to achieve a file locking workflow that plays as nicely with git as possible. Happy to provide more context on the reasoning / workflow at our studio, too. Once I can get the GVFS source building, I’d like to start poking around and see what this might actually entail regarding the design of GVFS.

[sanoursa]

This is a really great write up, thank you! I’ve followed through this same reasoning in the past and land on very similar thoughts, but I’ve also poked some holes in it that I haven’t found great solutions for. What are your thoughts on the following:

• Totality

I completely agree with you that locking is only effective if it’s reliable and complete. There’s nothing more frustrating than being told you can make your edits, spend hours on them, and then have to throw your work away because someone else beat you to it.

However, GVFS does not control the entry points into your repo. Ways in which you could get around any locking in GVFS:

1. Clone the repo with git and make whatever changes you want. It’ll take (much) longer to clone and run git commands, but once you’re past that hurdle, you can avoid dealing with the lock.
2. Clone your repo with GVFS, hydrate your files, disable GVFS - and make whatever edits you want.
3. Edit / upload files via the web. If you’re using any git service that allows web edits, that’s also a very easy way to get around the locks

For that reason, I keep coming to the conclusion that the only place that you can enforce a lock is at push time on the server. That would cover GVFS clone, git clone, and web edits, but it has a lot of implementation drawbacks as well - e.g. push-time policies are very expensive to implement.

• Best effort notifications

If we accept that locks can only be enforced at push time, that’s safe and correct, and very inconvenient for the artists who still end up wasting their time. So in addition to enforcement, we need best-effort notification of whether your edits will be successful or not. Your proposal for using GVFS to block edits if you can’t get a lock certainly helps there, but you would also need modifications to the web UI of whatever service you’re on… and maybe some way to do it in plain old git? Or just make the simplifying assumption that no one will be willing to wait hours for a clone just to get around a lock that they can’t push to the server anyway.

• Rebase

What happens if I’m currently at a commit that is a descendant of the most recent modification, I get the lock and make changes, and then I rebase my changes onto another commit that should not have been able to get the lock?

• Branches

I like your idea that you can only get the lock if you’re currently at a commit that is a descendant of the last modification. However, this assumes that all changes eventually flow back into master and that there are no other long lived branches. What if your workflow is that you create a release branch for every release, and only commit hotfixes into that branch moving forward, while master carries on? Now you need a notion of a lock per file per branch

• Good UX

Yes we can block writes from the file system. But how will an artist know that the write is failing because they don’t have the lock? And what if someone opens a file and works for hours without hitting Ctrl+S (gasp!) only to find that they can’t get the lock?

• Lots of questions about the “lock server”

These are mostly design/implementation details that can be worked out once you get past the issues above.

• Who holds the master information about the lock state of each file?
• What happens if a user obtains the lock and then never releases it? We could time out, but what if the timeout is 2 hours and some artist took 2 hours and 5 minutes to make their final edits? Do we need to capture every write event and let the server know we’re still busy? What will that do to the perf of our file system?
• What happens if someone wants to work offline and they’re willing to lose their work but the file system still won’t let them make edits because it can’t reach the lock server?

Overall, I do think file locking could be made to work, but I think GVFS would play only a small part in a bigger solution.

[sanoursa]

I think there are a lot of good ideas here, but it is diverging quite far from the purpose of GVFS, which is to make Git scale, but appear as if you’re working with normal Git on your normal file system. Right now GVFS does not add any features that Git doesn’t have, and does not force you to use any different Git commands than you normally would (outside of gvfs clone instead of git clone). In its ideal (which is not yet achieved, obviously), GVFS has zero observable behaviors of its own other than perf/scale improvements.

In general, what I would suggest is to first devise a locking scheme that works with normal Git at least one reference Git service. Once that is understood, it could definitely make sense to use GVFS as another enforcement tool, but I still don’t think GVFS is in the driver seat on this issue.

I’m going to close this issue because it’s not something I would see us pursuing as a GVFS feature unless it first becomes a Git feature. I’m more than happy to keep discussing it though, and see if we can land on a good solution.