Allow user to specify `index-url` during `kedro pipeline pull`

Description

Hey team.

Say I want to build up a collection of reusable pipelines. I decide to use a python package index as the distribution mechanism because I really like the versioning capabilities and how it integrates with my CI pipeline. But I don’t want to put my pipelines on PiPy. So I spin up a private python package repo and add its url under extra-index-url in my pip.conf. So far so good.

But the problem is that because pip doesn’t have a notion of index priority now the names of my pipelines potentially clash with all package names on PiPy. This is at best confusing and at worst a security nightmare.

If I was dealing with python packages, where one package depends on many others, I would either have to live with this or spin up something like devpi. But the cool thing about kedro pipelines is that I shouldn’t need to do that since kedro pipeline pull doesn’t require access to PiPy at all (IIUC). So what I’d love to do is restrict the pull command to my private index, only (no PiPy).

Possible Implementation

Since kedro pipeline pull uses pip download under the hood, I should be able to achieve what I want to do using pip’s --index-url argument. (I can already do this today by adding it to my global pip.conf but this messes with my ability to download other python packages so I don’t want to do that).

I can think of three ways this might work:

1. use shell=True in the subprocess.run call, here so that I can set PIP_CONFIG_FILE in my environment and it gets passed on to the pip download call
2. explicitly copy PIP_CONFIG_FILE from the environment that executes kedro pipeline pull to the subprocess.run call (env=...)
3. add an --index url argument to kedro pipeline pull and pass it through to pip download

I’m sure there’s other solutions, too. Maybe there’s one that already works with the current version?

Thanks for all your hard work!

[datajoely]

This is on the roadmap 😃 stay tuned

[merelcht]

Closing this for now. But feel free to re-open if this is still needed (cc: @willashford)