Handle Kubeflow authentication/authorization errors
See original GitHub issueDescribe the issue See below.
To Reproduce Steps to reproduce the behavior:
- Provision a secured KFP deployment
- Create a runtime configuration in Elyra that meets one of the following criteria
- no user name + password specified
- invalid user name specified
- invalid password specified
- Create a notebook
- Create a pipeline
- Add notebook to pipeline
- Using the “invalid” runtime configuration run pipeline OR export pipeline DSL and run DSL
- Observe failure
f"pipeline submitted: {api_endpoint}/#/runs/details/{run.id}", AttributeError: 'NoneType' object has no attribute 'id'
During a preliminary investigation I’ve noticed that there are several silent errors that are never detected. For example, experiment creation fails without raising an error. It appears that the kfp
client (tested with 0.7.0
) doesn’t seem to detect/handle/expose authentication/authorization issues caused by missing/invalid credentials.
Expected behavior A clear and concise description of what you expected to happen. Elyra analyzes the KFP response for signs that an API call might have failed even though no exception is raised or error returned.
Deployment information Describe what you’ve deployed and how:
- Elyra version: 2.3/master (likely also in earlier versions)
- Kubeflow 1.3 with security enabled (e.g. via DEX) - didn’t test earlier versions
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (5 by maintainers)
Top Results From Across the Web
KFP sdk client authentication error #4569 - kubeflow/pipelines
What steps did you take and what happened: Enabled authentication with Azure AD on AKS and installing Kubeflow with kfctl_istio_dex.v1.1.0.yaml ...
Read more >Connecting to Kubeflow Pipelines on Google Cloud using the ...
The following sections introduce how to authenticate your SDK ... Error “Failed to authorize with API resource references: there is no user ...
Read more >RBAC: access denied - error when accessing a Notebook ...
RBAC: access denied - error when accessing a Notebook Server as a ... rbac.authorization.k8s.io kind: ClusterRole name: kubeflow-edit ...
Read more >Authentication with Dex | Documentation - Charmed Kubeflow
Auth Flow. The authentication flow for an unauthenticated request consists of these steps: User makes an unauthenticated request. Redirected to /dex.
Read more >Planned Support for Kubeflow Pipeline Uploads via API?
However, when I try uploading a pipeline using the route "/apis/v1beta1/pipelines/upload" in PostMan, I get the following error:
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I believe we should treat this as a bug in Elyra because the call in https://github.com/elyra-ai/elyra/blob/master/elyra/pipeline/processor_kfp.py#L86-L88 returns
None
as session cookie when invalid credentials were specified. Unfortunately https://github.com/elyra-ai/elyra/blob/e82a3ec8b189ac94ade9d87e4482e0b760d5596b/elyra/pipeline/processor_kfp.py#L488 does not return a 403 HTTP status code if invalid credentials were provided. Therefore we might have to interpret the absence of a cookie as a failure condition and raise an authentication error.No guarantee because we are only testing with one Dex configuration. But since things are already failing should a host ever return an auth URL that our pattern match doesn’t catch, we are at least not regressing when we add the additional code(c: