[bug] Pipeline UI couldn't load mlpipline-ui-metadata (v1) with Minio gateway in Multi-user install on GCP

What steps did you take

I ran a simple 1 step pipeline, which aimed to display a webpage in visualization UI (the Output).

What happened:

In a KF cluster with MinIO gateway (GCS is the actual storage), Multi-users install (on GCP), the Visualization UI couldn’t display the webpage. Screenshot

What did you expect to happen:

In a KF cluster with in-cluster MinIO setup (a PD is the actual storage), Multi-users install (on GCP), the Visualization UI could display the webpage Screenshot

Environment:

Platform: GCP, IAP-ingress enabled, CloudSQL, MinIO gateway (with a GCS bucket)

• How do you deploy Kubeflow Pipelines (KFP)? A custom from kubeflow/gcp-blueprints, because of my company policies, CloudSQL/MySQL is connected via a private connection (VPC) instead of using cloudsql proxy. I also can’t use Cluster management.
• KFP version: 1.7.0
• KFP SDK version: kfp 1.8.4 kfp-pipeline-spec 0.1.12 kfp-server-api 1.7.0

Anything else you would like to add:

The Visualization UI looks like loading an artifact with namespace https://{host}/pipeline/artifacts/minio/om-ml-pipeline-np/artifacts/viz-sample-rtdtl/2021/10/17/viz-sample-rtdtl-2393626054/mlpipeline-ui-metadata.tgz?namespace=om

Clicking to the link I should get: Failed to get object in bucket om-ml-pipeline-np at path artifacts/viz-sample-rtdtl/2021/10/17/viz-sample-rtdtl-2393626054/mlpipeline-ui-metadata.tgz: S3Error: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Remove the ?namespace=om in the link then I can download the artifact.

I’ve already follow https://github.com/kubeflow/pipelines/issues/4649 , but v2 is still in Beta and lacks some features I need in v1, so I have to stick with v1.

Labels

/area frontend /area backend /area sdk

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[jacobmalmberg]

I’ve just found out that after creating a workspace (namespace), the ml-pipeline-ui-artifact doesn’t have any MinIO config (eg: MINIO_ACCESS_KEY, MINIO_SECRET_KEY, …). By (manually) adding some env into the ml-pipeline-ui-artifact deployments yaml, all the pipeline UI artifacts can be displayed.

Taking a look at this sync.py https://github.com/kubeflow/manifests/blob/master/apps/pipeline/upstream/base/installs/multi-user/pipelines-profile-controller/sync.py#L267

I don’t see configmaps or secrets, which are created by the script, get mounted into the ml-pipeline-ui-artifact deployment.

@zijianjoy could it be a bug or my setup is wrong?

Mounting the ENV vars is proposed in PR https://github.com/kubeflow/pipelines/pull/5864.

[zijianjoy]

As we released KFP 1.8.1, it should include the mentioned fix. Feel free to reopen if the issue persists.