Support publishing git dependencies to PyPI using vendoring

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

I noticed that when trying to upload a package with a Git dep to PyPI, I get an error:

HTTP Error 400: Invalid value for requires_dist. Error: Can't have direct dependency: 'example @ git+https://github.com/example/example.git@v0.1.5'

My research has told me that this is because PyPI does not support non-package dependencies.

I was thinking, could Poetry support publishing packages with Git deps by copying the the contents of the git repository at the provided ref into the source distribution, and rewriting imports if needed to make sure the package was imported from a local directory instead? Then the wheel could be built from that.

This way the Git dep would be bundled with the package’s code, and PyPI wouldn’t complain. I think this could be a great feature that would set Poetry apart.

[abn]

Any idea what that would look like?

There will be a few things to be considered when providing a solution.

First of which would be to identify if this is infact a “core” requirement or a plugin wrapping around something like https://github.com/pradyunsg/vendoring.

Either way, we will have to determine how vendoring happens. Pre-pending site directories? Or embedding in-package eg: foobar.vendors.x?. The latter would mean the code itself will need to explicitly call the embedded package - which is safer. However, this means, you cannot simply swap the vendored library for a non vendored one without having to update the imports (note vendoring already can handle this, and is already the default - the reverse is not pssible afaict). I am more inclined to be on the safe side with these things, because - you vendoring a package should not break some other package as a rule, exception being you are building an application (eg: poetry-core does this, and so does poetry’s get-poetry.py installer - latter being a consequence of gaps in python’s application distribtution ecosystem).

Then there is evaluating the consequence of doing this for package resolution. Does ithis mean that, if your package depends on A and B (vendored, lets call it B’), and A depends on B (non vendored) the depenency resolver treats B and B’ as separate packages? or the same but different versions? Both have consequences that can be problematic.

These are the first things that come to my mind.

To be clear, I am not proposing we do this. Just discussing the possibilities. I am always on the fence whether vendoring is a good thing or a bad thing and if poetry should promote it or discourage it. 😃

EDIT: Oh, also there is the conversation around how does a PEP 517 build handle this? Or is the scope of this limited to wheels built by poetry itself and not using poetry-core build backend?

[neersighted]

Closing as a specialized variant of #4583 as submodules + the feature proposed there make more sense. Poetry’s use of VCS/Git dependencies is of the PEP 508/standard-to-the-ecosystem style. If you want to include arbitrary code vendored in to your install/built artifacts and control the paths, that issue tracks it.

I don’t think we’ll ever overload the way Poetry defines Git dependencies to allow for the implicit vendoring requested in this issue – instead, submodules and #4583 are the best way to achieve this, in a way that is more explicit and flexible.