Please provide a way to install pipenv without using vendored dependencies

As discussed in #1885 it is very hard for downstream (i.e. Debian) maintainers to package pipenv when it provides almost all its dependencies in the vendor and patched directory.

Quoting myself from https://github.com/pypa/pipenv/issues/1885#issuecomment-377778859

My main concern is still security: If your dependencies also handle their dependencies by vendoring them (and theirs) you’ll end up with multiple versions of the same package inside your vendors directory (e.g. the requests package). If one of those packages needs an update you’ll have a hard time fixing all of them. For downstream maintainers (Debian, Suse, etc) it is even worse as we usually just update the problematic library and assume this bug to be fixed for all packages depending on it. Now we actively have to search for copies of that library everywhere.

So in order to make life for downstream package maintainers easier, please provide a way – similar to what pip does – to install pipenv “unbundeled” and thus using the packages installed on the target system.

[uranusjr]

Yes, I don’t object to your proposal of repurposing them to distinguish between different kinds of patching, my comment was merely exaplaining why they seem to lack clarity.

[jayvdb]

I’ve gone through the process of complete de-vendoring of pipenv/vendor, with only a few non-internet tests failing. Also see https://github.com/pypa/pipenv/issues/3630 for the tests which are failing probably due to not being declared as needing internet/bugs in the mocking. I am going through the process a second time now, hopefully cleaner, to avoid functional breakages caused by my first de-vendoring.

IMO the first step should be more clarity between pipenv/vendor vs pipenv/patched. There are quite a few packages in pipenv/vendor which are actually functionally patched, and IMO they should be moved to patched , or vendor and patched merged and the patches more clearly split into functional-changes-only and vendoring-only patches, ideally with functional-changes patches applied first (as upstream and de-vendoring needs them to cleanly apply to the upstream sdist), and vendoring patches based on top of the functional-changes-only patches where applicable.

And a concerted effort needs to be put into upstreaming the functional-changes patches. I’ve found several patches which havent been provided to upstream, and sometimes I couldnt find an issue raised upstream. Probably there are good reasons for this, but it work to be done. Also there are a few functional changes which obviously are not acceptable upstream, such as the sys.path.append at https://github.com/pypa/pipenv/blob/dc41e66/pipenv/vendor/pipdeptree.py#L17 , and as others have indicated those are hacks which need to be removed before they can be de-vendored normally, and some are probably really not easy to remove (and not a priority for the core devs). IMO people wanting to de-vendor are going to need to help tackling these problems.

Anyone interested can find all pipenv/vendor and some pipenv/patched packages as .spec files in https://build.opensuse.org/project/show/devel:languages:python , and https://build.opensuse.org/project/show/home:jayvdb:coala:python3-bears has some patches applied. A static copy of the messy first iteration of de-vendored pipenv .spec can find it at https://build.opensuse.org/package/show/home:jayvdb:branches:home:jayvdb:coala:python3-bears/python-pipenv

The following are my notes wrt the patches in the 2018.11.26 release - some of these are now irrelevant, but packagers of 2018.11.26 release will still benefit from them.

Most important is https://github.com/pypa/pipenv/commit/be3c7714ee374ad0311becccbe72fdbe54414767 , not a listed patch, replacing cursor with vistir due to recently uncovered legal problems with using GPL cursor in a non-GPL project.

vendor/patches:

• click-completion-enum-import.patch - vendoring only
• delegator-close-filehandles.patch - merged upstream in https://github.com/kennethreitz/delegator.py
• dotenv-windows-unicode.patch - platform win32 only, not likely relevant to de-vendor
• drop_scandir_import.patch - vendoring only: removing dso
• passa-close-session.patch - merged upstream in https://github.com/sarugaku/passa
• pip_shims_module_names.patch - vendoring only
• pipdeptree-updated-pip18.patch - not vendoring only - includes import path fix
• tomlkit-fix.patch - removal of typing to ease vendoring, and https://build.opensuse.org/package/view_file/home:jayvdb:coala:python3-bears/python-tomlkit/trivia-fix.patch ; https://github.com/sdispater/tomlkit/issues/45 raised
• vistir-imports.patch - vendoring only
• yaspin-signal-handling.patch - vendoring, adding colorama & cursor (c.f. note re cursor above), and https://build.opensuse.org/package/view_file/home:jayvdb:coala:python3-bears/python-yaspin/signal-handling.patch now mentioned upstream at https://github.com/pavdmyt/yaspin/issues/32 (appears to be windows specific, so not a concern for de-vendoring)

patched/patches:

• crayons.patch - extensive changes, some maybe breaking changes
• pip18.patch - patch to notpip internals
• pipfile.patch - several enhancements to https://github.com/pypa/pipfile , including https://github.com/pypa/pipfile/commit/04c2616b5fe90f35ef169635e82d332e363f1059 - no release since Feb 2017
• piptools.patch - enhancements, active discussion at https://github.com/jazzband/pip-tools/issues/679
• _post-pip-update-pep425tags.patch - patch to notpip internals
• _post-pip-update-pypi-uri.patch - patch to notpip vendored distlib: https://bitbucket.org/pypa/distlib/issues/120/pypiorg (merged upstream)
• _post-pip-update-requests-imports.patch - vendoring only, fixing notpip vendoring of requests

There are also a few other patches possibly needed by de-vendoring

• https://github.com/sarugaku/vistir/pull/61
• https://github.com/sarugaku/vistir/pull/62
• https://github.com/sarugaku/vistir/pull/68
• https://github.com/pypa/pip/pull/6319