cloud versioning: pull non-DVC cloud updates

In a project with a cloud-versioned remote, there could be either manual or automated updates happening on the remote outside of any DVC process. For example, a daily ETL process uploads new data directly to the cloud remote. Or someone unfamiliar with DVC wants to upload some new data.

A DVC user should be able to not only pull the versions of data tracked in their project, but also pull the latest version of data available on the cloud. For example, dvc update could be used to get the latest data the same way it does today for imports.

[dberenbaum]

One thing to clarify that may simplify this: For now, it’s fine to keep the current syntax for dvc update where it requires a target. We may need to support dvc update without a target eventually, but for now being able to update a dataset at a time is enough.

@dberenbaum one thing we didnt discuss was how to handle deletion for a standalone file output, where a tracked file no longer exists in the bucket (and has DELETE marker set).

If you import data with import/import-url, then delete the source and run dvc update, it will fail. Let’s stay consistent with that for now.

[pmrowla]

After discussion with @dberenbaum, scope for this issue on initial release will be

• dvc update will only be applicable in worktree = true scenarios (and not version_aware = true, worktree = false)
• update should pull the latest versions of outs we are aware of (and have .dvc files for locally).
  • For a file output, update would get latest modifications to the file
  • For a dir output, update would get latest modifications to files in the dir, newly added files in the dir, and deletions of files within the dir
• update will ignore files/dirs in the remote bucket that we are not aware of already (i.e. new files/dirs that are in the bucket but do not appear in any of our local .dvc/dvc.yaml files)