How to improve information provided to users when pip backtracks

(Taken over from #8714) @pradyunsg @brainwane @nlhkabu @georgiamoon @pfmoore @uranusjr

What is the issue?

During a pip install, pip needs to work out which version of a package is a good candidate to install. If it tries one version, finds out it isn’t compatible it needs to “go back” (backtrack) and download an older version, trying that, if it is successful it will continue onto the next package, if not it will continue to backtrack until it finds a compatible version, or it will eventually display “resolution impossible”.

If pip starts backtracking during dependency resolution, it does not know how long it will backtrack, and how much computation would be needed.

Why is this a bad experience?

pip does not communicate to the user what it is doing, why it is doing it, how long it will take, and what they can do. It can also take a very long time - “why is pip downloading multiple versions of the same package?”.

What do we need to do?

We need to communicate to the user: 1) what pip is doing, 2) why it is doing it , and 3) what the user can do, 4) at what verbosity level we should communicate what information at.

What happens right now?

pip does not give the user any information to understand it is backtracking, or why it is happening. pip does not keep track of the number of times it backtracks.

An example of current output:

(backtrack-pyrax) debian@vps-695a9d26:~/backtrack-pyrax$ pip install pyrax==1.9.8 --use-feature=2020-resolver Collecting pyrax==1.9.8 Downloading pyrax-1.9.8-py2.py3-none-any.whl (346 kB) |████████████████████████████████| 346 kB 10.4 MB/s Collecting python-novaclient==2.27.0 Downloading python_novaclient-2.27.0-py2.py3-none-any.whl (312 kB) |████████████████████████████████| 312 kB 19.2 MB/s Collecting PrettyTable<0.8,>=0.7 Downloading prettytable-0.7.2.zip (28 kB) Collecting iso8601>=0.1.9 Downloading iso8601-0.1.13-py3-none-any.whl (9.3 kB) Collecting python-keystoneclient>=1.6.0 Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB) |████████████████████████████████| 397 kB 28.2 MB/s Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB) |████████████████████████████████| 395 kB 27.0 MB/s Downloading python_keystoneclient-3.20.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 24.4 MB/s Downloading python_keystoneclient-3.19.1-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 21.3 MB/s Downloading python_keystoneclient-3.19.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 26.2 MB/s Downloading python_keystoneclient-3.18.0-py2.py3-none-any.whl (393 kB) |████████████████████████████████| 393 kB 22.1 MB/s Downloading python_keystoneclient-3.17.0-py2.py3-none-any.whl (382 kB) |████████████████████████████████| 382 kB 23.8 MB/s Downloading python_keystoneclient-3.16.0-py2.py3-none-any.whl (376 kB) |████████████████████████████████| 376 kB 27.5 MB/s Downloading python_keystoneclient-3.15.1-py2.py3-none-any.whl (385 kB) |████████████████████████████████| 385 kB 30.4 MB/s Downloading python_keystoneclient-3.15.0-py2.py3-none-any.whl (378 kB) |████████████████████████████████| 378 kB 21.4 MB/s Downloading python_keystoneclient-3.14.0-py2.py3-none-any.whl (372 kB) |████████████████████████████████| 372 kB 21.1 MB/s Downloading python_keystoneclient-3.13.1-py2.py3-none-any.whl (381 kB) |████████████████████████████████| 381 kB 21.8 MB/s Downloading python_keystoneclient-3.13.0-py2.py3-none-any.whl (374 kB) |████████████████████████████████| 374 kB 24.7 MB/s Downloading python_keystoneclient-3.12.0-py2.py3-none-any.whl (374 kB) |████████████████████████████████| 374 kB 22.1 MB/s Downloading python_keystoneclient-3.11.0-py2.py3-none-any.whl (372 kB) |████████████████████████████████| 372 kB 25.9 MB/s …etc…

What is wrong with this output?

Right now, pip tells the user nothing. If the user was to leave pip “do its thing” 2 outcomes are possible - pip might install all the packages, or it might fail due to not finding compatible packages. This might take a long time.

Do we need to do UX research?

We think it’s common during a pip install package-name command for pip to backtrack 5 or 6 times and more than 13 backtracks is uncommon. We don’t have any specific data.

We have created a poll asking users for a better idea of how often pip backtracks during an install.

What improvements can we make?

One way to improve this is to give pip the ability to keep track of how often it backtracks, and which candidates it backtracks on – using (backtracking(self, candidate) hook on resolvelib.

Using this, pip could then:

• record the backtracking choices
• count the number of backtracking choices

Proposed solution

We propose pip tells the user 1) why it’s is backtracking, and 2) what they need to/can do.

We propose pip: After the 1st backtrack pip prints: <message 1> After the 8th backtrack pip prints: <message 1> After the 13th backtrack pip prints: <message 1><message 2> Both messages are displayed at verbosity level 0.

Message 1 text

INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while.

Message 2 text

INFO: This is a complicated process. You might need to provide the
dependency resolver with stricter constraints to reduce runtime.
If you want to abort this run, you can press Ctrl + C to do so.
    To improve how pip performs, tell us why this happened here: https://github.com/pypa/pip/issues/XXXX

How it would look in a real output:

$ pip install pyrax==1.9.8 Collecting pyrax==1.9.8 Downloading pyrax-1.9.8-py2.py3-none-any.whl (346 kB) |████████████████████████████████| 346 kB 10.4 MB/s Collecting python-novaclient==2.27.0 Downloading python_novaclient-2.27.0-py2.py3-none-any.whl (312 kB) |████████████████████████████████| 312 kB 19.2 MB/s Collecting PrettyTable<0.8,>=0.7 Downloading prettytable-0.7.2.zip (28 kB) Collecting iso8601>=0.1.9 Downloading iso8601-0.1.13-py3-none-any.whl (9.3 kB) Collecting python-keystoneclient>=1.6.0 Downloading python_keystoneclient-3.22.0-py2.py3-none-any.whl (397 kB) |████████████████████████████████| 397 kB 28.2 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. Downloading python_keystoneclient-3.21.0-py2.py3-none-any.whl (395 kB) |████████████████████████████████| 395 kB 27.0 MB/s Downloading python_keystoneclient-3.20.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 24.4 MB/s Downloading python_keystoneclient-3.19.1-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 21.3 MB/s Downloading python_keystoneclient-3.19.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 26.2 MB/s Downloading python_keystoneclient-3.18.0-py2.py3-none-any.whl (393 kB) |████████████████████████████████| 393 kB 22.1 MB/s Downloading python_keystoneclient-3.17.0-py2.py3-none-any.whl (382 kB) |████████████████████████████████| 382 kB 23.8 MB/s Downloading python_keystoneclient-3.16.0-py2.py3-none-any.whl (376 kB) |████████████████████████████████| 376 kB 27.5 MB/s Downloading python_keystoneclient-3.15.1-py2.py3-none-any.whl (385 kB) |████████████████████████████████| 385 kB 30.4 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. Downloading python_keystoneclient-3.15.0-py2.py3-none-any.whl (378 kB) |████████████████████████████████| 378 kB 21.4 MB/s Downloading python_keystoneclient-3.14.0-py2.py3-none-any.whl (372 kB) |████████████████████████████████| 372 kB 21.1 MB/s Downloading python_keystoneclient-3.13.1-py2.py3-none-any.whl (381 kB) |████████████████████████████████| 381 kB 21.8 MB/s Downloading python_keystoneclient-3.13.0-py2.py3-none-any.whl (374 kB) |████████████████████████████████| 374 kB 24.7 MB/s Downloading python_keystoneclient-3.12.0-py2.py3-none-any.whl (374 kB) |████████████████████████████████| 374 kB 22.1 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. INFO: This is a complicated process. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us why this happened here: https://github.com/pypa/pip/issues/XXXX Downloading python_keystoneclient-3.11.0-py2.py3-none-any.whl (372 kB) |████████████████████████████████| 372 kB 25.9 MB/s

<install process continues>

Documentation

We’ll also add documentation to the user guide giving users an explanation of what backtracking is, why is happens, and possible ways to reduce it from occuring. Covered by #9039.

Better solution - using time

A better solution would be to tell the user how long this backtrack will take, and show progress.

There is no way to distingush between small number of steps on “big package downloaded twice” vs lots of steps of “small package downloaded manyyyy times”. And showing time progress might be very difficult to do.

Design rationale

The information here explains the UX design decisions we’ve made.

This is based on the “progressive disclosure” approach. It is a common UI design pattern. pip will give the user more information the longer the backtracking goes on, because it is more likely the user may want/need to abort the process.

Message 1 - provides the user with the right amount of information at the right time - after the first number of backtracks, pip tells the user why this is happening and it will take time.

Message 2 - More than likely, pip will not need to backtrack any more. If it does, it then provides the user with more information - this time about what they can do if they do not want it to continue.

If we include a link to a GH issue, users can provide information about the situations where backtracking exceeded 13 tries, allowing us to get a better idea of the necessary trigger points, and possibly improve the situation.

Related GH issues

https://github.com/pypa/pip/issues/8713 https://github.com/pypa/pip/issues/8683 https://github.com/pypa/pip/issues/8922 https://github.com/pypa/pip/issues/8883

How you can help

1. Message trigger points

If you’d like to help us improve the trigger points for message 1 and 2, you can complete this short poll (1 question!).

2. Message text

Are these messages acceptable? If you have suggestions, please give some explanation why.

3. Github issue link

We propose opening a new Github issue to track situations where message 2 is displayed. Using a Github issue template we can ask users specific questions.

Any other suggestions?

[pradyunsg]

The broader question of “how do we highlight the important parts of pip’s output” is probably a separate issue, though.

psst #4649.

[pfmoore]

Nice explanation of the issue! 🙂

One thought that occurred to me is that we should also try to ensure that the “progressive” messages don’t get lost in among other output - your “how it would look in real output” example has this problem. Maybe coloured output would help here, or careful use of indentation. The broader question of “how do we highlight the important parts of pip’s output” is probably a separate issue, though.