[frontend] S3Error: 503 in ml-pipeline-ui-artifact

How did you deploy Kubeflow Pipelines (KFP)?

We are attempting to deploy the kubeflow pipeline over a local cent-os cluster using kustomize

We use multi-user isolation

k8s v1.19.6

On a custom Kubernetes deployment. KFP version: 1.7.0 KFP SDK version: kfp 1.8.6 kfp-pipeline-spec 0.1.11 kfp-server-api 1.7.0

Steps to reproduce

For create run we use V2 compatible mode and default package_root

Try open/load/view any artefact in runs gui

Failed to get object in bucket mlpipeline at path v2/artifacts/pipeline/Georgy test/025b7274-585b-4aac-95c3-9ca2d609d47e/irs/output_dataset: S3Error: 503

Remove the ?namespace in the link then we can download the artefact.

log from ml-pipeline-ui-artifact when we try get artefact with ?namespace

GET /pipeline/artifacts/get?source=minio&peek=256&bucket=mlpipeline&key=v2%2Fartifacts%2Fpipeline%2FGeorgy+test%2F025b7274-585b-4aac-95c3-9ca2d609d47e%2Firs%2Fmarkdown_artifact
Getting storage artifact at: minio: mlpipeline/v2/artifacts/pipeline/Georgy test/025b7274-585b-4aac-95c3-9ca2d609d47e/irs/markdown_artifact
S3Error: 503
    at getError (/server/node_modules/minio/dist/main/transformers.js:138:15)
    at /server/node_modules/minio/dist/main/transformers.js:158:14
    at DestroyableTransform._flush (/server/node_modules/minio/dist/main/transformers.js:80:10)
    at DestroyableTransform.prefinish (/server/node_modules/readable-stream/lib/_stream_transform.js:138:10)
    at DestroyableTransform.emit (events.js:223:5)
    at prefinish (/server/node_modules/readable-stream/lib/_stream_writable.js:619:14)
    at finishMaybe (/server/node_modules/readable-stream/lib/_stream_writable.js:627:5)
    at endWritable (/server/node_modules/readable-stream/lib/_stream_writable.js:638:3)
    at DestroyableTransform.Writable.end (/server/node_modules/readable-stream/lib/_stream_writable.js:594:41)
    at IncomingMessage.onend (_stream_readable.js:693:10) {
  code: 'UnknownError',
  amzRequestid: null,
  amzId2: null,
  amzBucketRegion: null
}

log from ml-pipeline-ui when we try get artefact without ?namespace

GET /pipeline/artifacts/minio/mlpipeline/v2/artifacts/pipeline/Georgy%20test/025b7274-585b-4aac-95c3-9ca2d609d47e/irs/output_dataset
Getting storage artifact at: minio: mlpipeline/v2/artifacts/pipeline/Georgy test/025b7274-585b-4aac-95c3-9ca2d609d47e/irs/output_dataset

Materials and Reference

We’ll be grateful if you can get any clue what we can look more.

We tried to:

• env according #6750 and #4649

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[pwzhong]

@holadepo have you fixed the problem? I am facing the same issue. @lightning-like Could you explain a bit more what istio changes you made?

[RakeshRaj97]

@dvaldivia Thanks for that great tutorial. It is very helpful and saved me a ton of effort by centrally managing all data in one external minio. I highly appreciate and support your effort on making that tutorial and please continue to update it on a regular basis if possible.

MINIO_SSL=‘true’ in ml-pipeline-ui deployment still not solving the issue. My current image for ml-pipeline-ui is gcr.io/ml-pipeline/frontend:1.8.2

I also see this error

Failed to get object in bucket mlpipeline at path artifacts/create-pipeline-4sgrp/2022/10/18/create-pipeline-4sgrp-2895230044/main.log: Error: getaddrinfo ENOTFOUND internal-minio.example.com