Airflow constraints file constraints-1.10.7/constraints-3.7 incorrect constraints preventing install

Apache Airflow version: 1.10.7

Kubernetes version (if you are using kubernetes) (use kubectl version): N.A.

Environment:

• Cloud provider or hardware configuration: N.A.
• OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server 7.6 (Maipo)
• Kernel (e.g. uname -a): 3.10.0-957.35.2.el7.x86_64
• Install tools: pip, python3.7
• Others:

What happened: Could not install airflow 1.10.7 using constraint file for 1.10.7.

ERROR: Could not find a version that satisfies the requirement future<0.17,>=0.16.0 (from apache-airflow[celery,postgres,statsd])
ERROR: No matching distribution found for future<0.17,>=0.16.0

What you expected to happen: pip install success

The 1.10.7 constraint file (that was created using 1.10.10 dependencies) includes a constraint for future==0.18.2 while the setup.py for that version includes the requirement that future>=0.16.0, <0.17, which is incompatible. This can be considered part of https://github.com/apache/airflow/issues/12838, as this happens only when using the new pip 20.3 resolver, but I think pip is behaving correctly, and the real problem is in the constraint file.

How to reproduce it:

python3.7 -m venv venv_python37_airflow
cd venv_python37_airflow
. ./bin/activate
python3.7 -m pip install --upgrade pip
python3.7 -m pip install wheel
AIRFLOW_VERSION=1.10.7
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
python -m pip install "apache-airflow[celery,postgres,statsd]==${AIRFLOW_VERSION}" --constraint ${CONSTRAINT_URL}

Anything else we need to know: From my testing removing the constraint for future==0.18.2 solved this meaning it’s the only non-valid constraint for this version. I believe this happens in other earlier versions, but I did not test version earlier than 1.10.7.

[potiuk]

Thanks for reporting this. I will leave it for reference to anone that will try to install Airflow 1.10.7, however, our recommendation is to move forward with 1.10.14 (bridge release for 2.0) with the goal of moving to 2.0 as soon as possible. This is the release where constraints have been finally solved once and for all (also in the 2.0 version) and we highly recommend migrating to 2.0. 2.0.0rc2 is being voted and it is very likely the official 2.0.0 of Airflow will be released Monday/Tuesday. 1.10.14 is already released last week.

[potiuk]

While I don’t doubt that there are several problems with pip 20.3, as you documented in the attached issues, I don’t think this one falls under them. The constraints file for 1.10.7 is plainly and simply wrong, which is not surprising since it was created for 1.10.10. As I wrote above it includes a constraint for future==0.18.2 while the setup.py for that version includes the requirement for future>=0.16.0, <0.17, so any good resolver would indicate that there is a mismatch.

The problem is that before 20.3 PIP was happily accepting and resolving this anyway.

But I think there is nothing to fix now anyway. 1.10.7 is prehistory and we are recommending everyone to migrate to 1.10.14 which has good constraints (those constraints BTW do not work with pip 20.3 anyway). So similarly like with any other problems with previous releases - 1.10.14 + PIP 20.2.4 (for now) solves the problems we know about (including installation problems). I don’t think any other action is needed at this stage.