exp push: fails for >50MB commits

dvc exp push can fail since GitHub rejects commits >50MB in size. Perhaps use DVC cache instead for such cases?

Part of https://github.com/iterative/cml/issues/560

/CC @pmrowla

[pmrowla]

For reference, the issue here was discussed in yesterdays meeting:

When we generate a checkpoint commit, anything marked as a pipeline dependency or output that is also cache: false will be forcefully tracked in Git. This is needed to so that DVC can properly preserve/restore the state of the workspace when resuming checkpoint runs.

The problem w/the large commits is most likely occurring when users have large, intermediate cache: false outputs/deps that they don’t want tracked at all (by DVC or Git). But DVC cannot tell the difference between an output that is cache: false “because the user will track it with Git” or cache: false “because it should not be tracked it all”, and we end up incorrectly tracking these files in the checkpoint Git commits.

---

One thing to note here is that DVC will not do the forced tracking for files which are both cache: false and .gitignored. This issue is potentially also happening because users don’t bother gitignoring these intermediate files/dirs (since the user knows not to manually git add them). However this doesn’t work for an automated CI run like in CML.

If these intermediate files/dirs are properly gitignored, it would also stop DVC from generating these bloated checkpoint commits. I think we need to clearly document this behavior on both the DVC and CML sides.

@dberenbaum @casperdcl

[casperdcl]

we track every iteration

On a related note we may want to add an option to only keep the last N checkpoints. May save disk space as well as sanity when doing exp show on 10 billion checkpoints.