Katib cannot obtain trail task information after istio is enabled. After completion, the pod status is NotReady

/kind bug

What steps did you take and what happened: [A clear and concise description of what the bug is.] Since my training container needs to communicate with ML-pipeline, I need to enable istioto add Kubeflow-userid and the corresponding AuthorizationPolicy. However, automL cannot obtain the status information of Taial after istio is opened.Is there any way to solve this problem? Is there any other way to solve the communication problem between the training container and ml-pipeline？

@ewilderj @tmielika @mhausenblas @marcoceppi @pineking At present,I don’t open istio， I have added the corresponding request header in the source code, the following is the error message：

I1015 01:26:39.579167      54 main.go:136] kfp_server_api.exceptions.ApiException: (403)
I1015 01:26:39.579191      54 main.go:136] Reason: Forbidden
I1015 01:26:39.579198      54 main.go:136] HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Fri, 15 Oct 2021 01:26:39 GMT', 'server': 'istio-envoy', 'x-envoy-decorator-operation': 'ml-pipeline.kubeflow.svc.cluster.local:8888/*'})
I1015 01:26:39.579222      54 main.go:136] HTTP response body: RBAC: access denied

Thank you for your guidance

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

[anencore94]

Here’s the main reason of this issue and temporal solution if you need to use istio enable in trial: https://github.com/kubeflow/katib/issues/1638

You could build your own metrics-collector image with such code and use it.

[anencore94]

| I’m glad you replied to my question. I have solved the above problem in my own way, but thank you very much

Thanks for the reply ! If you are ok, how about share your solution? Your solution may help other people who suffer with this istio-stuff. @longpi1