Resolving Home Assistant takes 12 hours with new resolver

Description

The new resolver introduced in pip takes 12 hours to resolve Home Assistant, making it unusable for Home Assistant. Our current workaround is using the legacy resolver.

(Time measured on AMD Ryzen 7, 32GB memory)

The legacy resolver is going to be removed in #9631 making pip unusable for Home Assistant.

Home Assistant focuses on privacy and local control and is world’s largest open source home automation platform. In 2020 it was the 2nd most active Python project on GitHub (per State of the Octoverse 2020). We integrate over 1000 different APIs and each API is integrated via a Python package installed via pip (requirements.txt).

All our dependency versions are pinned, but the libraries that we use don’t pin their dependencies. This leads to long resolving time as the new resolver will download all the different potentially compatible versions to find the right one. Libraries don’t pin their dependencies because a library tries to be compatible with as many other projects as possible (we always make sure we don’t rely on HA specific API implementations).

I understand you don’t want to keep the legacy resolver around forever, but we would like to see that the new resolver is usable prior to it’s removal.

Expected behavior

Done in less than 15 minutes.

pip version

21.3.1

Python version

3.10

OS

Debian 11

How to Reproduce

git clone https://www.gitHub.com/home-assistant/core home-assistant-core
cd home-assistant-core
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements_all.txt

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[potiuk]

In the past we used our constraints file to “solve” conflicts. That is similar to what is requested in pypa/pipenv#4530 to work with the new resolver. Having that feature means we can move fast again with shipping security updates when needed without having to rush to ship a security fix and update X other packages that cause backtracking issues.

Yep - we have very sophisticated mechanism of both preparing and using constraints. In fact we are publishing a set of constraints for our users so that they can install airflow reliably. For example this is the only “official” way you can install 3.6 version of Airlfow 2.2.3 (from https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html)

pip install "apache-airflow[celery]==2.2.3" \
   --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.3/constraints-3.6.txt

The problem here is that it backtracked when we tried to upgrade the constraints. We have > 500 dependencies, so we do not upgrade constraints manually - we automated it. In our CI, the constraints will be automatically upgraded in the main build and whenever dependencies change. We are using --eager-upgrade for that. After the --eager-upgrade succeds, we run tests and only when tests pass we publish new constraints in “main” (which later are frozen when we release Airflow).

So in our case the backtracking happens (sometimes) when we attempt to generate new constraints with --eager-upgrade (which excludes using existing constraints effectively).

BTW. It’s pretty sophisticated mechanism with Airflow’s >500 deps - I gave recently presentation on that where you can see most of the why’s and how’s if you are interested: https://www.youtube.com/watch?v=_SjMdQLP30s&t=2549s

[notatallshaw]

Actually I rebuilt the requirements files with the above freeze information.

Using these and fixing/removing the constraints file you can install using pip 21.3.1. Though you will need to figure out which libraries are too old for your requirements and look at why they have conflicts by updating them and see what the conflicts are.

Here is a diff on the requirements_all.txt:

-PyRMVtransport==0.3.3
+PyRMVtransport==0.3.2
-aioambient==2021.11.0
+aioambient==2.0.0
-aiobotocore==1.2.2
+aiobotocore==2.1.0
-aioridwell==2021.12.2
+aioridwell==2021.10.0
-aiosteamist==0.3.1
+aiosteamist==0.1.0
-boto3==1.16.52
+boto3==1.20.24
-hole==0.7.0
+hole==0.6.0
-# homeassistant.components.mycroft
-mycroftapi==2.0
-open-meteo==0.2.1
+open-meteo==0.0.1
-py17track==2021.12.2
+py17track==2021.12.0
-pyicloud==0.10.2
+pyicloud==0.7.3
-pymodbus==2.5.3
+pymodbus==1.5.2
-pyoverkiz==1.1.0
+pyoverkiz==1.1.1
-pysml==0.0.6
+pysml==0.0.5
-# homeassistant.components.stiebel_eltron
-pystiebeleltron==0.0.1.dev2
-pytile==2021.12.0
+pytile==2021.10.0
-simplisafe-python==2021.12.2
+simplisafe-python==2021.11.2
-systembridge==2.2.3
+systembridge==1.1.5
-vallox-websocket-api==2.8.1
+vallox-websocket-api==2.6.0

And the diff on requirements.txt:

-aiohttp==3.8.1
+aiohttp==3.7.4.post0
-async_timeout==4.0.2
+async_timeout==3.0.1
-pip>=8.0.3,<20.3