Make install command upgrade packages by default

• Pip version: Future versions, hopefully 10.0
• Python version: All supported
• Operating System: All supported

Based on discussion over at #59, there’s interest in making the install command upgrade an installed package by default. This behaviour would make pip consistent with various other package managers, with regards to the behaviour of it’s install command.

This issue is meant to be the location for that discussion, since this deserves it’s own issue.

[dstufft]

Ok whew, I ignore this discussion for a few days it appears to have blown up on both the sig and here 😃

I’ve tried to read over what’s gone on in this thread, but well it’s long and information dense so I might miss something, however you’re about to get a wall of words.

---

I don’t believe it’s completely fair to say that the current behavior of pip install -U <foo> makes people more secure across the board. Yes, I can easily point out some projects like PyCrypto or cryptography where regressions are rare (particularly security regressions) and new releases generally include improvements to security. I think focusing only on those misses other cases though, such as the case where a new version of something on PyPI has caused a regression in security. There’s I think, two other cases though, both which can boil down to “upgrading or not doesn’t affect security at all” but differ in whether an upgrade is an OK thing to do for them or not.

Overall, I don’t think that recursive upgrades is a good security mechanism and if one of our goals is to prevent people from running old, insecure versions of software (and I think it should be a goal) than I think the way to achieve that is not to try and hang ourself off of the behavior of upgrades (and just pray and hope that they’ve happened to run an update in some amount of time) but to instead devote time to a dedicate solution to the problem. This may be something like pip list -o but which checks explicitly for security problems, it may be checking the entire installed set of packages against PyPI to see if there are any known security issues with any of them, or it may take on some completely other format. However, I think it’s important that this isn’t tied to some semi related functionality and that it actually covers the entire environment and not just whatever the user happens to be using. If someone does pip install requests[security] once, and then from there on out does pip install -U requests– we’re going to completely miss updates to pyopenssl and cryptogaphy and such if we only rely on recursive upgrades.

So pushing aside the security concerns for a moment, I think we need to take a look at what behaviors are most likely to give people what they want. Unfortunately with an ecosystem as large and with as varying use cases as Python I suspect there is no singular answer to “what people want”. There are a few interrelated behaviors being discussed here, so let’s tackle them one at a time.

For pip install --upgrade <foo>, we have evidence to that fact that our current behavior is actively harmful, so much to the point that projects are going out of their way lie to pip to prevent triggering that behavior. I think that we can all agree that something where people feel the need to actively subvert (not only in their own projects, but also advocate to other projects) likely need some refinements to how it actually works. In this case the only real solution is to attempt to avoid upgrading (or downgrading!) where possible, and to prefer the already installed version, unless the user has explicitly asked for that to be changed or we can’t satisfy the version constraints otherwise. I can’t see any other reasonable way to implement this that isn’t going to accidentally trigger 30+ minute builds which possibly result in a version that is less suitable for the task at hand (not using an optimized BLAS or something).

Advocating for leaving the current behavior of pip install --upgrade as it is, is essentially advocating against projects that depend on numpy from being honest about whether or not they depend on numpy. If someone has another suggestion for how we might solve the numpy problem [1] then I think they should bring it up.

I know that one suggestion has been adding a --non-recursive-upgrade flag or a sort of --upgrade-strategy flag, but I think that these ideas largely serve only to complicate the mental model people have of pip. For the vast majority of packages (particularly pure python ones that don’t have a security sensitive role) it’s not going to matter a whole lot whether we upgrade them or not, upgrading is low cost but there’s little downside to keeping them pinned to the installed version (unless the person finds a reason, a feature or a bug to explicitly upgrade that package). However we’re living in the edge cases here and I only really see the two that matters, hard-to-upgrade packages and security sensitive packages, and like I mentioned above I think that if we’re going to be worrying about ensuring folks get security upgrades we need a real mechanism for that, not a half hearted hope that an important upgrade got caught in a recursive upgrade at some point. Given that I think we need dedicated support for security sensitive, for most packages this won’t matter, and for the hard-to-upgrade case there’s only really one answer, I think that shunting this behind it’s own flag is a bad idea without justification for the ongoing cost of maintaining a whole option for this [2]. I also think that the more conservative approach has to be the most obvious approach or we don’t really solve things for the hard-to-upgrade crowd, so if we did add a new option, we’d still want to change the behavior of --upgrade by default and add in some explicit option to get the less conversative/safe approach to upgrades.

I don’t believe we should allow a package to be able to mark itself for eager (or non eager) upgrades. I think this is something that needs to be consistent amongst all packages for end users to be able to have any hope of having a reasonable mental model of what pip is going to do to their system when they execute some command. That being said, I could see us adding the ability to have people mark versions insecure on PyPI and warn people if they have an insecure version installed on their system (or at the very least, if they’re about to install one).

So, now that we’ve covered --upgrade, the other inter-related issue here is what should we do with pip install versus pip install --upgrade. Personally I think that we should make the two mean the same thing, HOWEVER it might be more reasonable to focus the discussion first on just the behavior of --upgrade and leave pip install alone for now. Once we get the solution to upgrading sorted out we can tackle what we’ll do about pip install itself.

[1] Although to be completely honest it’s not strictly related to Numpy. I’ve seen people break their systems or their installs time and time again because of an inadvertent ugprade. It’s true that a lot of project’s don’t have correct lower bounds, but I believe it’s equally true (or more so) that they don’t have correct upper bounds either. One particularly important thing is that it’s possible to determine what the correct lower bounds are at time of packaging, but it’s impossible to determine what the correct upper bounds are.

[2] Options are not free, they incur a cost and it’s important to attempt to reduce the number of them you have as much as you can. While you typically can’t reduce them to zero, a pattern I see far too much in OSS software is the desire to please everyone by adding more and more options, when really it’s just a mechanism for avoiding making a decision that may be unpopular with some group of people.

[rgommers]

pip gets used in 3 different contexts IME:

• maintaining production environments
• building fresh environments in a reproducible way - e.g. container builds
• testing

@rbtcollins I’m not clear on whether your definition of “production environment” includes everything from “new user who installed Python + some packages” to “advanced users with multiple venvs” to “production deployments runnings apps/websites”? Your comments about security seem like you’re mostly worried about the last category, but imho that is the least interesting one because it caters to the most knowledgeable users. Defaults should be chosen for non-expert users, the ones who installed Python on their laptop/desktop and want to get their analysis done or website to work.