Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[backend] persistence agent don't update pipeline run status after workflow deleted

See original GitHub issue

Environment

How did you deploy Kubeflow Pipelines (KFP)?

KFP version: 1.5.0

KFP SDK version:

Steps to reproduce

launch a pipeline run, and observe the status of nodes in the run as expected from UI.
use command kubectl delete workflow xxx to delete the workflow corresponding to the pipeline run, then the unfinished nodes in the pipeline run stay running, and never been updated.
the delete workflow may happen when we delete the scheduledworkflow, and then the workflow without the parent scheduledworkflow got GC

Expected result

The running nodes in the pipeline run been marked as terminated

Materials and Reference

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

cc @Jeffwan

Issue Analytics

State:
Created 2 years ago
Comments:7 (5 by maintainers)

Top GitHub Comments

2reactions

Jeffwancommented, May 27, 2021

My rough idea to improve would be that, persistence agent should also periodically list all runs from KFP API that are still in running state, if any of the runs do not have a corresponding argo workflow. We should update the DB record to maybe a special state called “WORKFLOW_DELETED” etc.

An alternative could be persistent agent watches workflow changes and if there’s a delete workflow event, check running state of associated run and update DB. I prefer to choose the way which can reduce apiserver pressure. 😄

1reaction

DaleSincommented, May 27, 2021

Maybe just list runs which have not been updated/reported for a certain time, it should be a few runs.