perf: remote indexes

As we discussed making dvc fast should be high priority as poor performance can draw people away easily. The big part of todays slowness is working with remotes, which almost always includes collecting file statuses, which could be slow for bigger remotes. All this leads to some form of indexes.

However, our remotes don’t provide a luxury of atomic group writes nor reads nor read-modify-write operations. We still can use the following strategy:

• make an index dir on remote, say index,
• write a list of files (with names, checksums, mtimes and/or file sizes) to an index file in that dir say 1.idx
• later when some client needs to update that, e.g. after pushing some new files it:
  • reads current index,
  • updates it,
  • writes new list to 2.<uuid>.idx,
  • removes 1.idx. This way in a case of a race we will have several index files.
• if a client needs to read an index it downloads all index files and combines them.

Since we will not only have adds, but also deletes we will need smart combine procudure like in CRDTs.

File format to be discussed, simple JSON or gzipped JSON with a list of files may do the job though.

What do you guys think? @shcheklein @dmpetrov @efiop @pared @mroutis

[Suor]

A note on garbage collection.

Since git has a distributed nature and we are collecting everything not referenced we may remove something recently pushed and only referenced in git commits not available locally yet - not pulled or not even pushed yet by an author.

The sane method to circumvent this is providing a grace period: do not gc anything newer than N days. In the case someone has just pushed something one shouldn’t and wants to remove that it should be still possible to do that:

dvc gc -c --grace-period=0

This is “I know what I am doing even thoufgh I just messed up flag” 😃

The grace period might simplify some conflict resolution above. E.g. we might use normal listing in gc instead of index listing to collect orphane files. This should be discussed in #2325.

[efiop]

We’ll create a ticket in dvc-objects/data for this kind of odb indexing with a summary and in the mid term and will close this.