epic: CML <> DVCLive

Make it easy to publish CML reports from DVCLive.

cml comment create --publish --watch=1min report.md &
python train.py --dvclive-report=report.md

• #1026
  • cml comment {create,update} --publish: auto-publish images embedded in markdown reports
  • cml comment {create,update} --watch file-watcher
    • file-based mutex lock
    • upstream: DVCLive support for file-based mutex lock
• cml publish seed to help with overwriting outdated files
  • potentially related to #291
• upstream: DVCLive option to write md instead of html
  • https://github.com/iterative/dvclive/issues/91
  • https://github.com/iterative/dvc-render/issues/62

future follow-up

• Happier userflow path
  • user specifying DVCLive output formats to be md (or md and html)
  • DVCLive auto-detecting CI env to decide what format to output
  • CML auto-converting html2md (via e.g. pandoc or some JS lib)
• Studio renders reports (rather than just linking to GH)
  • detects dvclive reports (maybe with a special watermark) as a flag to poll for updates every minute

[casperdcl]

@daavoo soon (after #1073 <- #762), the idea is:

# simple
cml report watch --publish report.md &
python dvclive_script.py --output=report.md

# synchronised alternative
cml report watch --publish --trigger-file=/tmp/dvclive report.md &
python dvclive_script.py --output-report=report.md --output-trigger=/tmp/dvclive

where the “synchronised alternative” flow avoids potential contention. TL;DR pseudocode:

DVCLive

def next_step():
    ...
    if output_report:
        if output_trigger and output_trigger != output_report:
            if not output_trigger.exists():
                mkreport(output)
                output_trigger.touch()
        else:
            mkreport(output)

CML

while True:  # watcher thread
    if triggerFile and triggerFile != reportFile:
        if exists(triggerFile):
            upload(reportFile, ignore_missing=True)
            rm(triggerFile)
    else:
        upload(reportFile, ignore_missing=True, only_if_changed=True)

[daavoo]

🔔 @daavoo with regard to #1036 (comment)

I think I prefer the current approach in the P.R.

What would be the main benefit of this alternative? The ability to share env vars with the child process?

What happens when integrating with DVC cml whatever dvc repro?

CML could trigger comment updates when the child process outputs a magic string to the standard output/error

This feels a little more clunky to me than the file watcher approach.