client.list_experiments() not working as well as creating runs, authentication failing

What steps did you take

We deployed kubeflow on premise using kustomize file https://github.com/kubeflow/manifests/tree/v1.3.1, v 1.3.1 Created a notebook, tried to create a run from it using kfp

I can’t connect from the notebook to pipelines (to create runs and see experiments ) import kfp kfp_endpoint=None client.list_experiments()

but i can enumerate pipelines
import kfp kfp_endpoint=None client = kfp.Client(host=kfp_endpoint) print(client.list_pipelines())

What happened:

i’am getting Reason: Internal Server Error HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Wed, 13 Oct 2021 11:48:57 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'}) HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines .....

I tried multiple things, some poeple faced the same issue, adding the namespace, creting a manifest as described in https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/#connect-to-kubeflow-pipelines-from-the-same-cluster Nothing worked for me . I would really appreciate any help

What did you expect to happen:

See the expierments, create runs from my pipelines

Environment:

• How do you deploy Kubeflow Pipelines (KFP)?

Deployed using Kustomize https://github.com/kubeflow/manifests/tree/v1.3.1

• KFP version:

• KFP SDK version:

Anything else you would like to add:

if i pass the YRL of the pipeline UI in kfp client, there is no error but nothing is returned {'experiments': None, 'next_page_token': None, 'total_size': None}

Labels

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[nschenone]

@rexad In my case, the issue was an outdated version of kfp. After running pip install --upgrade kfp and upgrading to kfp==1.8.6 my issue was resolved via the method @zijianjoy mentioned in the e2e notebook example.

[MarkusThill]

I would also like to highlight this point: After trying many things for hours I also just did a (I imagined somehow that I already did this; apparently I did not): !pip install --upgrade kfp before running (dont forget to restart your kernel…)

import kfp
client = kfp.Client()
print(client.list_experiments())

So @zijianjoy 's suggestion actually works with the newest kfp version…

Would you like to try the multi-user mode approach here? https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/#multi-user-mode