Pipelines: current working directory behavior is inconsistent

While working on https://github.com/elyra-ai/elyra/issues/1033 I’ve noticed that the current working directory is different for local execution and KFP execution.

Consider the following pipeline:

• notebook.ipynb creates a file in cwd, say file.txt
• subdir/notebook2.ipynb creates a file in cwd, say file2.txt

Results:

• Running the pipeline locally file.txt is created in the directory where notebook.ipynb is located (as expected)
• Running the pipeline locally file2.txt is created in the subdir directory where notebook2.ipynb is located (as expected)
• Running the pipeline on KFP file.txt is created in the directory where notebook.ipynb is located (as expected)
• Running the pipeline on KFP file2.txt is created in the parent directory of subdir (fail)

The cwd location needs to be consistent across runtimes: the directory where the script | notebook is stored or the location of the pipeline file.

[lresende]

After more tests, I believe this is actually not an issue per si, but more a side effect on how we are executing pipeline nodes in KFP, where we collect all artifacts related to a node and then untar them on the root to be executed (in this case, losing the notion that it is contained in a subdirectory during execution) but all is properly captured/propagated. I will investigate it further and discuss more details/side effects of this with @ptitzler but for now, I am not going to block the release because of this.

[ptitzler]

Are you still intending to work on this @lresende or can the assignment be removed?