[bug] With Python SDK, TFJob won't stop running

A TFJob would not stop running even after the training code exits.

tfjob_client.get_job_status returns ‘Running’. tfjob_client.wait_for_job timeouts With, tfjob_client.get_logs broken(#1182), running kubectl -n ${NAMESPACE} logs ${JOB_NAME}-worker-0 -c tensorflow gives training logs, from which it can be understood that training is over. But the job does not stop.

Running tfjob_client.is_job_succeeded gives False

[jinchihe]

@yashjakhotiya Thanks the deep checking. how about the status by kubectl get tfjob ${JOB_NAME} -n ${NAMESPACE}. I think the Python API should be consistent with backend CLI output, otherwise that’s bug.

[yashjakhotiya]

Turned off istio for TFJob pods with

template=V1PodTemplateSpec(
        metadata=V1ObjectMeta(
            annotations={'sidecar.istio.io/inject':'false'}
        ),
        spec=V1PodSpec(
            containers=[container]
        )
    )

The TFJob stopped as expected. Turns out the istio sidecar running has caused problems for this as well as the logs issue (#1182). Closing this issue now.