ScheduledWorkflow continues to try to sync with disabled recurring run workflow

What steps did you take:

I created a recurring run on the KFP UI. I let it run for some time, then disable it.

What happened:

When I view the logs of the ml-pipeline-scheduledworkflow, I notice the WorkflowScheduler continues to attempt to sync with the ml-pipeline pod on behalf of the now disabled recurring run. It produces the following logs:

"Submitting workflow for ScheduledWorkflow (aip2682catchuponjjjjj): nothing to submit (next scheduled at: 2020-11-18 22:11:02 +0000 UTC)" ScheduledWorkflow=aip2682catchuponjjjjj

When I delete the recurring run configuration, this no longer happens (as expected).

What did you expect to happen:

The ScheduledWorkflow pod should not attempt to sync for a recurring run that’s been disabled. It produces a lot of logs because we currently have a lot of recurring runs that we disable and no longer care about, but they still produce a lot of logs.

Environment:

KFP v.1.0.0

How did you deploy Kubeflow Pipelines (KFP)?

KFP version: 1.0.0 KFP SDK version:

aip-kfp-sdk              1.0.0+38064f78 
kfp                      1.0.0          
kfp-pipeline-spec        0.1.2          
kfp-server-api           1.0.1  

Anything else you would like to add:

I initially thought this was a Catchup (backfilling issue). The ScheduledWorkflow continues to sync with a recurring run with catchup turned on. When catched is turned off, it should no longer sync (but it still does).

/kind bug

[alexlatchford]

Thanks for pointing this out @Bobgy, we’ll see how our mitigation works and whether we need to tackle this too if it doesn’t, thanks for being transparent on priorities. Will reach out if we have any more questions!

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment “/reopen” to reopen it.