Unable to undeploy app after deployment with failing healthcheck

Hello, I’m running SCDF on k8s with helm and found issue looking to me as a bug.

• k8s version - 1.16 (AWS EKS eks.1)
• helm chart - https://hub.helm.sh/charts/stable/spring-cloud-data-flow/2.7.1

Steps to reproduce:

1. Create simplest stream with time and log : “time | log”
2. Change liveness-probe-path and readiness-probe-path to value that intentionally breaks healthcheck for time app.
3. Update stream.
4. Initial version of time app is destroyed and new one is started. After couple of minutes app’s state is changed to ‘failed’ and stream’s status to ‘partial’ (you can try to perform this step couple of times to generate more failing app versions). 5*. If at this stage you try to undeploy the stream or destroy it both deployments for time and log apps are deleted - this is expected (don’t try to remove stream when trying to reproduce the issue, I mentioned this step only to show that deletion of the stream at this stage works and SCDF can delete failing app).
5. For initial stream existing since step 1 change values for liveness-probe-path and readiness-probe-path back for time app to normal. New version of time is started.
6. Previous version is still trying to start, running into crashloopbackoff.
7. Now destroy, or undeploy the stream. Deployments of healthy apps are deleted, but failing ones are preserved. Also tried stream all destroy --force - same results.

If multiple updates are made and app cannot start, all these versions are preserved after stream is destroyed, also occupying resources of k8s cluster. Only manual deletion with kubectl delete deployment appDeploymentName helps.

In skipper logs I can also observe such lines:

2020-07-13 13:40:57.974 INFO 1 — [eTaskExecutor-2] o.s.c.s.s.d.s.HandleHealthCheckStep : Release testTimeLogStream-v2 has been DEPLOYED 2020-07-13 13:40:57.974 INFO 1 — [eTaskExecutor-2] o.s.c.s.s.d.s.HandleHealthCheckStep : Apps in release testTimeLogStream-v2 are healthy. 2020-07-13 13:40:57.984 INFO 1 — [eTaskExecutor-2] o.s.c.s.s.d.s.HandleHealthCheckStep : Deleting changed applications from existing release testTimeLogStream-v1 2020-07-13 13:40:57.995 WARN 1 — [eTaskExecutor-2] o.s.c.s.s.d.strategies.DeleteStep : For Release name testTimeLogStream, did not undeploy existing app time as its status is not ‘deployed’.

So it looks like it’s expected situation with current DeleteStep logic. However to me it looks like bug, as there are not cleaned resources left in k8s but from SCDF perspective stream’s health gets back to healthy.

Let me know if I need to provide more details. But setup is pretty basic and the issue can be reproduced easily.

[jvalkeal]

Better have these as a separate issues vs. #966

[tsimur-abayeu]

Hey, @jvalkeal ! Do I understand correctly, that the fix most likely will be delivered in 2.7.0.M1? As #972 and #975 mention it.