shouldn't pipenv rather depend on the packages in vendor than providing them directly?

Hello,

I’m currently packaging pipenv for Debian and I wonder what’s the purpose of the vendor directory and its contents? I understand the purpose of patched but shouldn’t pipenv rather depend on those packages in vendor (e.g. via install_requires in setup py) than providing them?

From a packaging point of view this is also very hard as I’ll probably have to un-bundle the vendor packages one by one and make pipenv depend on the corresponding Debian packages.

Assuming the packages in vendor are unpatched, would you accept pull requests that try to un-bundle those packages in vendor?

[ncoghlan]

As @gsemet notes, the key problem for tools like pip/pipenv/setuptools/etc is that they’re part of bootstrapping the dependency management system, and we can’t necessarily assume that that is already working in an end users’ environment, just because they have a Python install.

Debian & Ubuntu, for example, ship without a working ensurepip and venv by default - users need admin privileges to get things working (see https://github.com/pypa/python-packaging-user-guide/issues/399 ).

Fedora ships a working ensurepip and venv, but they do so by relying on pip’s dependency bundling at the virtual environment level and then keeping the pip package itself up to date.

The Mac OS X system Python is infamously broken by default (in more ways than just not supporting pip), as Apple decided to head down the Swift path instead.

pip and setuptools have the extra problem of actually ending up in the run and/or build environments of the projects being developed, giving them an extra incentive to hide their dependencies, whereas pipenv is mainly avoiding polluting the user level site-packages directory. However, having a random upgrade in the user site-packages directory break pipenv would also be a major problem when it came to keeping pipenv managed environments up to date, so there’s a significant resilience concern there as well.

So the answer to @venthur’s original question that started the issue is “No, it shouldn’t, as doing so would make bootstrapping more difficult, and introduce a greater chance of an end user accidentally breaking their ability to update their pipenv managed environments in a way they can’t readily recover from”.

I’m going to close this particular issue on that basis - if folks wanted to open to open separate issues for switching to pip-style vendoring (with clear dependency declarations, and automated updates for vendored dependencies), and then configuring pyup.io to monitor those dependencies for new releases, that would be a good thing.

More generally, I’d ask that folks avoid making the assumption that we aren’t already well aware of the risks and trade-offs between fully integrated systems (fewer security updates due to the use of shared components, but either requiring more coordinated integration testing before an update is released, or else increasing the risks of failure on end user systems due to untested combinations), and isolated application silos (multiple app updates needed for the same security update, but each app can be developed and tested independently without a complex distro-style coordination layer).

The key design assumption that pipenv makes that a lot of other developer tools don’t is that we expect a large proportion of our user base to be students, educators, and research scientists that are looking to write and run their first Python applications. We default them to running the latest version of everything, and require them to explicitly opt out and pin old versions if that’s what they want to do instead, but we mostly assume that they’re writing software for themselves.

Even for more experienced application developers, we mostly assume that they’re either working on deployment to hosted PaaS environments (so dependency updates are just a git push away), app store application development (so they have to bundle anything beyond the base platform APIs regardless), or else they’re using pipenv to manage the dependencies for their test suite and development environment, rather than for the deployed component.

This is going to be irritating for folks that prefer to treat their systems as carefully crafted interlocked webs of dependencies, and don’t want to delegate responsibility for any part of that system to anyone else. However, even the traditional Linux operating system developers have conceded that the “giant ball of mud” approach to operating system development breaks down once you reach the level of thousands or tens of thousands of intertwined components, and are switching to technologies like Kubernetes, Flatpak, and Snappy to help break that scaling barrier, while mitigating the harm that can be caused by outdated components in application bundles that haven’t updated appropriately.

[venthur]

Well, then let’s open another issue requesting the pip-style vendoring, shall we?