question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Handle Kubeflow authentication/authorization errors

See original GitHub issue

Describe the issue See below.

To Reproduce Steps to reproduce the behavior:

  1. Provision a secured KFP deployment
  2. Create a runtime configuration in Elyra that meets one of the following criteria
    • no user name + password specified
    • invalid user name specified
    • invalid password specified
  3. Create a notebook
  4. Create a pipeline
  5. Add notebook to pipeline
  6. Using the “invalid” runtime configuration run pipeline OR export pipeline DSL and run DSL
  7. Observe failure
    f"pipeline submitted: {api_endpoint}/#/runs/details/{run.id}",
     AttributeError: 'NoneType' object has no attribute 'id'
    

During a preliminary investigation I’ve noticed that there are several silent errors that are never detected. For example, experiment creation fails without raising an error. It appears that the kfp client (tested with 0.7.0) doesn’t seem to detect/handle/expose authentication/authorization issues caused by missing/invalid credentials.

Expected behavior A clear and concise description of what you expected to happen. Elyra analyzes the KFP response for signs that an API call might have failed even though no exception is raised or error returned.

Deployment information Describe what you’ve deployed and how:

  • Elyra version: 2.3/master (likely also in earlier versions)
  • Kubeflow 1.3 with security enabled (e.g. via DEX) - didn’t test earlier versions

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
ptitzlercommented, May 4, 2021

I believe we should treat this as a bug in Elyra because the call in https://github.com/elyra-ai/elyra/blob/master/elyra/pipeline/processor_kfp.py#L86-L88 returns None as session cookie when invalid credentials were specified. Unfortunately https://github.com/elyra-ai/elyra/blob/e82a3ec8b189ac94ade9d87e4482e0b760d5596b/elyra/pipeline/processor_kfp.py#L488 does not return a 403 HTTP status code if invalid credentials were provided. Therefore we might have to interpret the absence of a cookie as a failure condition and raise an authentication error.

0reactions
ptitzlercommented, May 4, 2021

No guarantee because we are only testing with one Dex configuration. But since things are already failing should a host ever return an auth URL that our pattern match doesn’t catch, we are at least not regressing when we add the additional code(c:

Read more comments on GitHub >

github_iconTop Results From Across the Web

KFP sdk client authentication error #4569 - kubeflow/pipelines
What steps did you take and what happened: Enabled authentication with Azure AD on AKS and installing Kubeflow with kfctl_istio_dex.v1.1.0.yaml ...
Read more >
Connecting to Kubeflow Pipelines on Google Cloud using the ...
The following sections introduce how to authenticate your SDK ... Error “Failed to authorize with API resource references: there is no user ...
Read more >
RBAC: access denied - error when accessing a Notebook ...
RBAC: access denied - error when accessing a Notebook Server as a ... rbac.authorization.k8s.io kind: ClusterRole name: kubeflow-edit ...
Read more >
Authentication with Dex | Documentation - Charmed Kubeflow
Auth Flow. The authentication flow for an unauthenticated request consists of these steps: User makes an unauthenticated request. Redirected to /dex.
Read more >
Planned Support for Kubeflow Pipeline Uploads via API?
However, when I try uploading a pipeline using the route "/apis/v1beta1/pipelines/upload" in PostMan, I get the following error:
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found