constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2

Apache Airflow version: 1.10.11

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment: Running command on macos.

What happened:

I am trying to build a production-like image for our environment using following command:

./breeze build-image --production-image --python 2.7 --additional-extras "aws, ssh, vertica" --install-airflow-version="1.10.11"

However, I get the following error which seems to be due to pandas==0.25.3 though pandas==0.24.2 I think is the last version that supports 2.7.

Thus it appears to be a problem with the constraints file.

ERROR: Could not find a version that satisfies the requirement pandas==0.25.3 (from -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.11/constraints-2.7.txt (line 206)) (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19.1, 0.19.2, 0.20.0rc1, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0rc1, 0.21.0, 0.21.1, 0.22.0, 0.23.0rc2, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0rc1, 0.24.0, 0.24.1, 0.24.2)
ERROR: No matching distribution found for pandas==0.25.3 (from -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.11/constraints-2.7.txt (line 206))

What you expected to happen:

Successful build of image

How to reproduce it:

See breeze build-image command in description.

[ccage-simp]

Hi jarek -

I think I’m going to close this one. I noticed that the constraints-1-10 branch is already updated with the fixes I needed here ( pandas==0.24.2, etc.). Only difference is I was requesting old 1.10.11 version.

It helped me to learn how to provide my own version of constraints file to breeze command. I think it’s good enough for my needs. And now we are closer to upgrading to 1.10.14 with py3 and will then be ready for 2.x.

As for a PR for other updates to breeze scripts to allow using a local constraints file more easily, I do not have good generalized solution yet. If I do create that, I will try to submit my first AF pr.

Thanks for your help!

Chris

[potiuk]

I am confused why I need this constraints file to install all these packages that we don’t use in our production airflow instance. Maybe I should be starting with empty constraints? Or creating our own requirements.txt for packages we actually install in our prod airflow…

Constraints are not requirements. They are simply hint to PIP. They basicaly mean “if you are installing this package, then use this version”. No more no less. So it’s good to have constraint with ALL possible packages. But then you install only what you need.

OK, got it working by referencing constraints file from path ./docker-context-files/constraints-2.7.txt which is accessible inside docker.

Yeah. That was exactly supposed to be like this.

scripts/docker/install_airflow_from_latest_master.sh and scripts/docker/install_airflow.sh having different working path context and yet both try to use relative constraints path. I guess I’ll have to modify the scripts for now.

Would be great if you contribute those changes back 😃. happy to review them 😃