Do we need to change `kedro package`?

This is originated from an internal request.

Problem Statement

The original use case is that user will do kedro install and reference this package in Kedro’s pipeline with import some_package/something, which works fine locally. However, when it comes to deployment, kedro package throws error and does not respect the requirement file.

Kedro Version: 0.17

# fs_requirements.txt
/packages/some_package/something

• kedro install would install this package - with a relative/local directory. (Not a problem as it is deprecated since 0.18)
• `kedro package would not respect this syntax and throw error immediately.

Questions

1. Does the relative/local path works for pip install or python setup.py at all?
2. It’s probably a good idea to stay away from executing setup.py directly. kedro package is basically python setup.py clean --all bdist_egg. See https://github.com/kedro-org/kedro/issues/1724
3. If 1 works, how should kedro package handle it?

Notes: It’s not an issue ready for tech design/backlog yet, but feel free to comment on this as I think we need to gather some more evidence to understand the problem, or how packaging should works in general.

kedro install is just a thin wrapper of pip install -r src/requirements.txt with some extra piptools compile step

[AntonyMilneQB]

In case it’s not clear to anyone reading: kedro install doesn’t exist since kedro 0.18.0, but above it’s basically the same as pip install -r src/requirements.txt.

I’m 99% sure this is the same as what’s discussed in https://github.com/kedro-org/kedro/issues/1536. Essentially it boils down to “what do you do with non-packageable requirements?”.

• in the case of a local wheel file, there’s a workaround: package @ file:///path/to/package.whl
• in other cases, it doesn’t really make sense to try and package it as part of the project because you wouldn’t be able to pip install the requirement after deployment if the package isn’t available
• there’s a more general point that the way we handle setup.py vs. requirements.txt is basically wrong. But I don’t think this is going to change soon (unless e.g. we move to poetry and/or pyproject.toml for dependencies) because it does seem to be more common this way

[noklam]

@sinha1 Thank you for your reply. We do understand this inconsistency - however, in this case, it’s more of a Python packaging issue and not much Kedro can do to support this.

I agree it would be better if we throw some warnings & docs when people use this, but as you have noticed kedro install was removed since 0.18.0, so it’s no longer a problem.

I hope this makes sense, I am closing this issue down as we don’t have a plan to support this in the near future