Using softlinks for pointers to libraries

Created from discussion topic https://github.com/elyra-ai/elyra/discussions/1646 at request of @lresende

I’d like to have a directory setup something similar to the following:

root
  -- common_libs
  -- pipe_1
     -- pipe1_files
     -- softlink_to_../common_libs
  -- pipe_2 
     -- pipe2_files
     -- softlink_to_../common_libs

The idea is that there is just one common_libs directory containing all my common libraries, and each pipeline dir contains a soflink to that directory. That way, I can easily import the common libs into the stages of my pipeline.

The problem arises when I try to run the pipeline on kubeflow - the stages fail when trying to import anything from common_libs because it cannot find the file.

It appears the tar command used to create the archive for the pipeline stages includes the softlink to common_libs rather than the files themselves. This is the default behaviour of tar but I believe it can be changed by using the --dereference flag - which includes a copy of the file in the archive rather than the link.

Any chance of changing the tar command for building the stage archives to include the dereference flag ?

[simon-streamotion]

Let me check with my org re. contributions to open source code 😃

[ptitzler]

@simon-streamotion would you be interested in delivering the change in a pull request?