DVC feature requests

Collection of DVC issues which CML functionality needs

Potentially needs

• pulling & pushing cache (For syncing experiments. Any other reasons?)
  • https://github.com/iterative/dvc/issues/4268
  • retrieve plots from run-cache https://github.com/iterative/dvc/issues/4096
• dvc verify https://github.com/iterative/dvc/issues/5369
• dulwich auto-auth using CI config
• CI runner timeout
  • DVC handling SIGINT, SIGTERM, or SIGKILL mid-exp run and mid-checkpoint
    • often (always?) dvc.lock won’t be generated https://github.com/iterative/dvc/issues/6180
  • run-cache storage (e.g. Azure https://github.com/iterative/dvc/issues/5899)
• DVC needs to be aware of total number of checkpoints expected per experiment https://github.com/iterative/dvc/issues/6183
  • dvc exp run && dvc exp run should only execute once
  • interrupting dvc exp run followed by calling dvc exp run again should resume (rather than start from checkpoint 0)
• pulling & pushing cache (For syncing experiments. Any other reasons?)
  • fetch experiment cache data https://github.com/iterative/dvc/issues/4649, https://github.com/iterative/dvc/issues/4268
  • dvc exp push for >50MB commits (e.g. somehow push to DVC remote rather than Git remote?) https://github.com/iterative/dvc/issues/6181

Needs

• CI runner timeout
  • CML re-provisioning runner #208, #174
  • attached storage #161
  • dvc exp push upon each checkpoint (e.g. via user callback? Or builtin option https://github.com/iterative/dvc/issues/6182?)
• ~~DVC needs to be aware of total number of checkpoints expected per experiment https://github.com/iterative/dvc/issues/6183~~
  • or just insist that the user code must work out the current checkpoint number by looking at the workspace state

[courentin]

My 2 cents: I feel this issue https://github.com/iterative/dvc/issues/5369 is related to cml, I’d love to have the ability to check if I need to repro the pipeline without having to spin up a self-hosted runner and pull the data.

[casperdcl]

potential workflow:

dvc exp run --name JOB_ID_UNCHANGED_UPON_KILL_AND_RESTART --pull
# ... (auto)kill via SIGINT after ~72h ... # CML does this 5 min early
dvc exp push # CML does this
dvc push # CML does this
# ... CML restarts the workflow

better alternative:

dvc exp run --name CI_JOB_ID_UNCHANGED_UPON_KILL_AND_RESTART --pull --push-every-checkpoint
# ... (auto)kill via SIGINT after ~72h ...
# ... CML restarts the workflow

Note that using COMMIT_SHA instead of CI_JOB_ID might not work in cases where the exp params are not stored in the commit (i.e. 2 job ids with different params but yet same commit sha).