[frontend] logs and artifacts stored in different Minio locations and can't be accessed by frontent

Environment

• How did you deploy Kubeflow Pipelines (KFP)? from repo
• KFP version: 1.7.0 To find the version number, See version number shows on bottom of KFP UI left sidenav. -->
• KFP SDK version:

kfp                                   1.7.0
kfp-pipeline-spec                     0.1.9
kfp-server-api                        1.7.0

Steps to reproduce

• run a pipeline and store an artifact
• delete the pod

Then we see that:

• The artifact (and its preview) can be seen in the output list of Input/Output tab
• The metrics can be accessed by the visualisations tab
• A wrong link to the metrics is found in the outputs list of the input/Outputs tab
  • details read Failed to get object in bucket mlpipeline at path v2/artifacts/pipeline/example pipeline/e154ea68-d59c-46ba-aea4-ead5b9eb0ea8/square/metrics: S3Error: The specified key does not exist.
• The logs cannot be retrieved from minio in the logs tab
  • details read Error response: Could not get main container logs: Error: Unable to find pod log archive information from workflow status.

Looking in MiniO it seems

• the Artifact is stored at mlpipeline/v2/artifacts/pipeline/example%20pipeline/e154ea68-d59c-46ba-aea4-ead5b9eb0ea8/square/

• and the logs and metrics are stored at mlpipeline/artifacts/example-pipeline-ljzls/2021/08/25/example-pipeline-ljzls-1034021739/

minimal code sample:

import kfp
from kfp.v2.dsl import pipeline, component, Output, Metrics, Artifact


@component(base_image='python:3')
def square(x: int, metrics: Output[Metrics], data: Output[Artifact]):
    metrics.log_metric('test_metrics', 1)
    metrics.log_metric(f'{x} squared', x**2)
    print(f'{x} squared is {x**2}')
    
    with open(data.path, 'w') as f:
        f.write("some content")


@pipeline(name='example pipeline')
def pipeline():
    square(4)

client = kfp.Client(host='http://localhost:8081')
client.create_run_from_pipeline_func(
    pipeline, arguments={}, run_name='Example run',
    experiment_name='Default', enable_caching=False,
    mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE
)

Expected result

The logs and metrics can be retrieved from MiniO by the frontend even when pods are deleted

Materials and Reference

Is this a configuration issue? Should I move things to point only to v2 path?

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[jonasdebeukelaer]

Not sure if related to that issue @Bobgy but looking in workflow-controller-configmap-patch.yaml the keyFormat is completely different to where the UI looks for logs/metrics:

keyFormat: "artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}"

I tried changing this to

keyFormat: "v2/artifacts/pipeline/{{workflow.annotations.pipelines.kubeflow.org/run_name}}/{{workflow.labels.pipeline/runid}}"

but the object path wrong as UI looks for e.g. {{component display name}}/metrics whereas this saves as {{component display name}}-metrics.tgz

[capoolebugchat]

It has been one year, did anyone found out how to resolve/workaround this bug that’d be great, Im sure this is one of the major bugs.