Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

HTTP response body: RBAC: access denied from kfp_server_api when creating a tfx pipeline

See original GitHub issue

/kind question

What steps did you take:

Ran the below code:

What happened:

I got the below:

HTTP response body: RBAC: access denied
ERROR:root:Failed to get healthz info attempt 4 of 5.
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp/_client.py", line 312, in get_kfp_healthz
    response = self._healthz_api.get_healthz()
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/healthz_service_api.py", line 77, in get_healthz
    return self.get_healthz_with_http_info(**kwargs)  # noqa: E501
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api/healthz_service_api.py", line 162, in get_healthz_with_http_info
    collection_formats=collection_formats)
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 383, in call_api
    _preload_content, _request_timeout, _host)
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 202, in __call_api
    raise e
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 199, in __call_api
    _request_timeout=_request_timeout)
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py", line 407, in request
    headers=headers)
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/rest.py", line 248, in GET
    query_params=query_params)
  File "/home/jovyan/.local/lib/python3.6/site-packages/kfp_server_api/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kfp_server_api.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Tue, 12 Jan 2021 17:25:02 GMT', 'server': 'envoy', 'x-envoy-upstream-service-time': '0'})
HTTP response body: RBAC: access denied

What did you expect to happen:

Being able to create the pipeline

Environment:

Kubeflow 1.2, Jupyter Notebook

How did you deploy Kubeflow Pipelines (KFP)?

KFP version: build version v1beta1

KFP SDK version: kfp 1.3.0 kfp-pipeline-spec 0.1.3.1 kfp-server-api 1.3.0

Anything else you would like to add:

I’m using kubeflow in my username’s namespace

Issue Analytics

State:
Created 3 years ago
Comments:10 (2 by maintainers)

Top GitHub Comments

1reaction

alexcpncommented, Aug 12, 2021

For new clusters please see this comemnt from the issue 4440

https://github.com/kubeflow/pipelines/issues/4440#issuecomment-871927424

cat << EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: bind-ml-pipeline-nb-kubeflow-user-example-com
 namespace: kubeflow
spec:
 selector:
   matchLabels:
     app: ml-pipeline
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/kubeflow-user-example-com/sa/default-editor"]
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-header
  namespace: kubeflow-user-example-com
spec:
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      context: SIDECAR_OUTBOUND
      routeConfiguration:
        vhost:
          name: ml-pipeline.kubeflow.svc.cluster.local:8888
          route:
            name: default
    patch:
      operation: MERGE
      value:
        request_headers_to_add:
        - append: true
          header:
            key: kubeflow-userid
            value: user@example.com
  workloadSelector:
    labels:
      notebook-name: test2
EOF

In my notebook


import kfp
client = kfp.Client()
print(client.list_experiments())

Output

{'experiments': [{'created_at': datetime.datetime(2021, 8, 12, 9, 14, 20, tzinfo=tzlocal()),
                  'description': None,
                  'id': 'b2e552e5-3324-483a-8ec8-b32894f49281',
                  'name': 'test',
                  'resource_references': [{'key': {'id': 'kubeflow-user-example-com',
                                                   'type': 'NAMESPACE'},
                                           'name': None,
                                           'relationship': 'OWNER'}],
                  'storage_state': 'STORAGESTATE_AVAILABLE'}],
 'next_page_token': None,
 'total_size': 1}

0reactions

Bobgycommented, Mar 5, 2021

Duplicate of #4440