Change the cos directory layout for pipeline artifacts

Is your feature request related to a problem? Please describe. Spawned from https://github.com/elyra-ai/elyra/pull/1732

Describe the solution you’d like A clear and concise description of what you want to happen.

Currently Elyra uses the following directory layout for input and output artifacts that are stored in a cloud storage bucket:

<pipeline-name-with-timestamp>/
<pipeline-name-with-timestamp>/<node-1-archive>.tgz    # input artifacts for node 1
<pipeline-name-with-timestamp>/<node-2-archive>.tgz    # input artifacts for node 2
<pipeline-name-with-timestamp>/<output-artifact-1>
<pipeline-name-with-timestamp>/<output-artifact-2>
<pipeline-name-with-timestamp>/<output-artifact-3>
...

There are two issues with this layout:

• input and output artifacts are stored in the same location
• output artifacts can be overwritten if the pipeline is run multiple times outside of Elyra (e.g. on KFP or AA)

To resolve the issues we could

• separate input and output artifacts
• associate output artifacts with a “run id”

For example:

<pipeline-name-with-timestamp>/
<pipeline-name-with-timestamp>/<node-1-archive>.tgz
<pipeline-name-with-timestamp>/<node-2-archive>.tgz
<pipeline-name-with-timestamp>/<run-1-id>/<output-artifact-1>
<pipeline-name-with-timestamp>/<run-1-id>/<output-artifact-2>
<pipeline-name-with-timestamp>/<run-1-id>/<output-artifact-3>
<pipeline-name-with-timestamp>/<run-2-id>/<output-artifact-1>
<pipeline-name-with-timestamp>/<run-2-id>/<output-artifact-2>
<pipeline-name-with-timestamp>/<run-2-id>/<output-artifact-3>
...

Ideally this unique “run id” is something that’s meaningful to the user.

[ptitzler]

I don’t think we want to entertain having to open the pipeline files to determine their ‘scope’. If we need that reverse lookup (which is natural) this might require more thought.

I agree. We can always improve this incrementally and start by separating input and output artifacts. While such a change would impact the user workflow a bit, it doesn’t have any impact on any of the Elyra tooling/UI. A new release would be good timing to make such a change.

[kevin-bates]

RIght. Perhaps it might be better to treat it purely as a classifier or scope comes to mind. However, I don’t think we want to entertain having to open the pipeline files to determine their ‘scope’. If we need that reverse lookup (which is natural) this might require more thought.