AIRFLOW_CONSTRAINTS_REFERENCE does not work properly while building the Docker image

Apache Airflow version: 2.0.1

• OS: Linux (tested both on a GitLab pipeline and using Ubuntu 20.10)
• Kernel : 5.8.0-50-generic

What happened:

docker build does not work if --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2" is specified as a build arg on v2-0-stable branch:

docker build --pull --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2" --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --build-arg PYTHON_MAJOR_MINOR_VERSION="3.7" --build-arg AIRFLOW_VERSION="2.0.2" --build-arg AIRFLOW_INSTALL_VERSION="==2.0.2" -t airflow:2.0.2 .

The returned error is:

ERROR: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-2.0-3.7.txt

Sending build context to Docker daemon 31.52MB Step 1/164 : ARG AIRFLOW_VERSION="2.0.1" Step 2/164 : ARG AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv" Step 3/164 : ARG ADDITIONAL_AIRFLOW_EXTRAS="" Step 4/164 : ARG ADDITIONAL_PYTHON_DEPS="" Step 5/164 : ARG AIRFLOW_HOME=/opt/airflow Step 6/164 : ARG AIRFLOW_UID="50000" Step 7/164 : ARG AIRFLOW_GID="50000" Step 8/164 : ARG CASS_DRIVER_BUILD_CONCURRENCY="8" Step 9/164 : ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster" Step 10/164 : ARG AIRFLOW_PIP_VERSION=20.2.4 Step 11/164 : ARG PIP_PROGRESS_BAR="on" Step 12/164 : FROM ${PYTHON_BASE_IMAGE} as airflow-build-image 3.7-slim-buster: Pulling from library/python Digest: sha256:f6bcc6d1fe0efe4ff1fb5590c23fddbae36c8757921d5d89da4e2b24d0ae704a Status: Image is up to date for python:3.7-slim-buster ---> 30a42f143b4e Step 13/164 : SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"] ---> Using cache ---> 073cc7b1d93f Step 14/164 : ARG PYTHON_BASE_IMAGE ---> Using cache ---> 07fd5a8530bd Step 15/164 : ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} ---> Using cache ---> 6ad7ba8c32c6 Step 16/164 : ENV DEBIAN_FRONTEND=noninteractive LANGUAGE=C.UTF-8 LANG=C.UTF-8 LC_ALL=C.UTF-8 LC_CTYPE=C.UTF-8 LC_MESSAGES=C.UTF-8 ---> Using cache ---> 90946c667620 Step 17/164 : RUN apt-get update && apt-get install -y --no-install-recommends curl gnupg2 && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* ---> Using cache ---> df99afd34b26 Step 18/164 : ARG DEV_APT_DEPS=" apt-transport-https apt-utils build-essential ca-certificates gnupg dirmngr freetds-bin freetds-dev gosu krb5-user ldap-utils libffi-dev libkrb5-dev libldap2-dev libpq-dev libsasl2-2 libsasl2-dev libsasl2-modules libssl-dev locales lsb-release nodejs openssh-client postgresql-client python-selinux sasl2-bin software-properties-common sqlite3 sudo unixodbc unixodbc-dev yarn" ---> Using cache ---> 74bfa6455df1 Step 19/164 : ENV DEV_APT_DEPS=${DEV_APT_DEPS} ---> Using cache ---> 419bd22d689e Step 20/164 : ARG ADDITIONAL_DEV_APT_DEPS="" ---> Using cache ---> 1f69197ad8b8 Step 21/164 : ENV ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} ---> Using cache ---> d3401572cd99 Step 22/164 : ARG DEV_APT_COMMAND=" curl --fail --location https://deb.nodesource.com/setup_10.x | bash - && curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - > /dev/null && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list" ---> Using cache ---> 5453be9d6ea3 Step 23/164 : ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} ---> Using cache ---> d2a07de63d88 Step 24/164 : ARG ADDITIONAL_DEV_APT_COMMAND="echo" ---> Using cache ---> 3ba10c8fed2d Step 25/164 : ENV ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND} ---> Using cache ---> 203a950784fc Step 26/164 : ARG ADDITIONAL_DEV_APT_ENV="" ---> Using cache ---> c24d0ff0e145 Step 27/164 : RUN mkdir -pv /usr/share/man/man1 && mkdir -pv /usr/share/man/man7 && export ${ADDITIONAL_DEV_APT_ENV?} && bash -o pipefail -e -u -x -c "${DEV_APT_COMMAND}" && bash -o pipefail -e -u -x -c "${ADDITIONAL_DEV_APT_COMMAND}" && apt-get update && apt-get install -y --no-install-recommends ${DEV_APT_DEPS} ${ADDITIONAL_DEV_APT_DEPS} && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* ---> Using cache ---> 8ddec6404e9a Step 28/164 : ARG INSTALL_MYSQL_CLIENT="true" ---> Using cache ---> ef368f06324b Step 29/164 : ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} ---> Using cache ---> 99dd0db02c95 Step 30/164 : COPY scripts/docker/install_mysql.sh /scripts/docker/install_mysql.sh ---> Using cache ---> 6b57562736d8 Step 31/164 : COPY docker-context-files /docker-context-files ---> Using cache ---> 3fa61dcc27a1 Step 32/164 : RUN bash ./scripts/docker/install_mysql.sh dev ---> Using cache ---> 4a39ffd3eb95 Step 33/164 : ARG AIRFLOW_REPO=apache/airflow ---> Using cache ---> 1ce430cea37b Step 34/164 : ENV AIRFLOW_REPO=${AIRFLOW_REPO} ---> Using cache ---> 0f1ea52fb56b Step 35/164 : ARG AIRFLOW_BRANCH=master ---> Using cache ---> 0ab86ebd84bd Step 36/164 : ENV AIRFLOW_BRANCH=${AIRFLOW_BRANCH} ---> Using cache ---> 1653bf37a629 Step 37/164 : ARG AIRFLOW_EXTRAS ---> Using cache ---> cfc2a7d5b8d2 Step 38/164 : ARG ADDITIONAL_AIRFLOW_EXTRAS="" ---> Using cache ---> 64d10026c456 Step 39/164 : ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} ---> Using cache ---> 94b7c449d1f4 Step 40/164 : ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow" ---> Using cache ---> a4c42604db01 Step 41/164 : ENV CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} ---> Using cache ---> b096c4c56bf6 Step 42/164 : ARG AIRFLOW_CONSTRAINTS="constraints-2.0" ---> Using cache ---> d934685366ce Step 43/164 : ENV AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} ---> Using cache ---> 4463555e3108 Step 44/164 : ARG AIRFLOW_CONSTRAINTS_REFERENCE="" ---> Using cache ---> 28e3663aa67a Step 45/164 : ENV AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} ---> Using cache ---> f742d6fd13f6 Step 46/164 : ARG AIRFLOW_CONSTRAINTS_LOCATION="" ---> Using cache ---> b492816e07dd Step 47/164 : ENV AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} ---> Using cache ---> b6357a41e012 Step 48/164 : ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-master" ---> Using cache ---> 9bfc76bc345d Step 49/164 : ENV DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} ---> Using cache ---> e3a8a75d1efc Step 50/164 : ENV PATH=${PATH}:/root/.local/bin ---> Using cache ---> 4251b29f4a77 Step 51/164 : RUN mkdir -p /root/.local/bin ---> Using cache ---> 8e8cdfdd911b Step 52/164 : RUN if [[ -f /docker-context-files/.pypirc ]]; then cp /docker-context-files/.pypirc /root/.pypirc; fi ---> Using cache ---> f719c4209ef5 Step 53/164 : ARG AIRFLOW_PIP_VERSION ---> Using cache ---> c3cd71821ce9 Step 54/164 : ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} ---> Using cache ---> 0d9fe5d7d839 Step 55/164 : ARG PIP_PROGRESS_BAR ---> Using cache ---> c8d7d0ff4112 Step 56/164 : ENV PIP_PROGRESS_BAR=${PIP_PROGRESS_BAR} ---> Using cache ---> f53a45e612be Step 57/164 : ENV AIRFLOW_INSTALL_USER_FLAG="--user" ---> Using cache ---> aef348201641 Step 58/164 : ENV AIRFLOW_INSTALL_EDITABLE_FLAG="" ---> Using cache ---> 7ad5cdc9a28b Step 59/164 : RUN pip install --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}" ---> Using cache ---> 7c1542ace7ea Step 60/164 : ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" ---> Using cache ---> ec98972f93b3 Step 61/164 : ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} ---> Using cache ---> 87dc43479c44 Step 62/164 : ARG INSTALL_PROVIDERS_FROM_SOURCES="false" ---> Using cache ---> a54b8079b5b2 Step 63/164 : ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES} ---> Using cache ---> 7748f5159b2b Step 64/164 : ARG AIRFLOW_VERSION ---> Using cache ---> fc451eb5c5ec Step 65/164 : ENV AIRFLOW_VERSION=${AIRFLOW_VERSION} ---> Using cache ---> 962e8db3b320 Step 66/164 : ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow" ---> Using cache ---> 9c327e191a12 Step 67/164 : ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD} ---> Using cache ---> 7fcc8a7e35d1 Step 68/164 : ARG AIRFLOW_VERSION_SPECIFICATION="" ---> Using cache ---> 65c0cef85e83 Step 69/164 : ENV AIRFLOW_VERSION_SPECIFICATION=${AIRFLOW_VERSION_SPECIFICATION} ---> Using cache ---> d71a809b3bab Step 70/164 : COPY scripts/docker/common.sh /scripts/docker/common.sh ---> cce345b407b5 Step 71/164 : COPY scripts/docker/install_airflow_from_branch_tip.sh /scripts/docker/install_airflow_from_branch_tip.sh ---> 90a9d7a2dbf6 Step 72/164 : ARG UPGRADE_TO_NEWER_DEPENDENCIES="false" ---> Running in 351a03c5eadb Removing intermediate container 351a03c5eadb ---> 2a1e3d73796e Step 73/164 : ENV UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES} ---> Running in 7f276dceff28 Removing intermediate container 7f276dceff28 ---> aa223dd2ed3c Step 74/164 : RUN if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && ${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then bash /scripts/docker/install_airflow_from_branch_tip.sh; fi ---> Running in 072dfa8d3e3c + [[ false == \t\r\u\e ]] Removing intermediate container 072dfa8d3e3c ---> 9ffda1fd10a1 Step 75/164 : ARG AIRFLOW_SOURCES_FROM="empty" ---> Running in 3ad0497283dc Removing intermediate container 3ad0497283dc ---> 6240b8996415 Step 76/164 : ENV AIRFLOW_SOURCES_FROM=${AIRFLOW_SOURCES_FROM} ---> Running in 5135b01b14aa Removing intermediate container 5135b01b14aa ---> 9d08170562ed Step 77/164 : ARG AIRFLOW_SOURCES_TO="/empty" ---> Running in ee5f63bab546 Removing intermediate container ee5f63bab546 ---> 1b96d150d9a4 Step 78/164 : ENV AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES_TO} ---> Running in 8a78f1a01f39 Removing intermediate container 8a78f1a01f39 ---> b12c53e9e969 Step 79/164 : COPY ${AIRFLOW_SOURCES_FROM} ${AIRFLOW_SOURCES_TO} ---> 25f3f9c5a9c0 Step 80/164 : ARG CASS_DRIVER_BUILD_CONCURRENCY ---> Running in df1afcd9d6eb Removing intermediate container df1afcd9d6eb ---> f5d602b5cd39 Step 81/164 : ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY} ---> Running in ae8b9d677f86 Removing intermediate container ae8b9d677f86 ---> 175e9317ccbf Step 82/164 : ARG ADDITIONAL_PYTHON_DEPS="" ---> Running in 67835e4a2067 Removing intermediate container 67835e4a2067 ---> caa7d1fd5d36 Step 83/164 : ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS} ---> Running in 4cf6636e86fc Removing intermediate container 4cf6636e86fc ---> 02b8b71b491d Step 84/164 : ARG INSTALL_FROM_DOCKER_CONTEXT_FILES="" ---> Running in a48e6e74a34a Removing intermediate container a48e6e74a34a ---> 07781f6dbe16 Step 85/164 : ENV INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES} ---> Running in 5ee6968fcb42 Removing intermediate container 5ee6968fcb42 ---> 34704d3dc4df Step 86/164 : ARG INSTALL_FROM_PYPI="true" ---> Running in 3ed6e67b7218 Removing intermediate container 3ed6e67b7218 ---> b818018f22b7 Step 87/164 : ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI} ---> Running in 1c34feaf15b5 Removing intermediate container 1c34feaf15b5 ---> 85aba433d76c Step 88/164 : ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3" ---> Running in 4164e06f93cc Removing intermediate container 4164e06f93cc ---> 70ab1d79aeb8 Step 89/164 : WORKDIR /opt/airflow ---> Running in 0d1865dda571 Removing intermediate container 0d1865dda571 ---> 3f9d122e9a5e Step 90/164 : ARG CONTINUE_ON_PIP_CHECK_FAILURE="false" ---> Running in 8f84390bb706 Removing intermediate container 8f84390bb706 ---> 95dc9d5fe166 Step 91/164 : COPY scripts/docker/install*.sh /scripts/docker/ ---> acd7e23c242a Step 92/164 : RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then bash /scripts/docker/install_from_docker_context_files.sh; elif [[ ${INSTALL_FROM_PYPI} == "true" ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true ---> Running in 0583608402ee + [[ '' == \t\r\u\e ]] + [[ true == \t\r\u\e ]] + bash /scripts/docker/install_airflow.sh ++ set -euo pipefail ++ test -v INSTALL_MYSQL_CLIENT ++ test -v AIRFLOW_INSTALL_USER_FLAG ++ test -v AIRFLOW_REPO ++ test -v AIRFLOW_BRANCH ++ test -v AIRFLOW_PIP_VERSION ++ set -x ++ set +x

Installing all packages with constraints and upgrade if needed

ERROR: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-2.0-3.7.txt WARNING: You are using pip version 20.2.4; however, version 21.0.1 is available. You should consider upgrading via the ‘/usr/local/bin/python -m pip install --upgrade pip’ command. The command ‘/bin/bash -o pipefail -e -u -x -c if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == “true” ]]; then bash /scripts/docker/install_from_docker_context_files.sh; elif [[ ${INSTALL_FROM_PYPI} == “true” ]]; then bash /scripts/docker/install_airflow.sh; fi; if [[ -n “${ADDITIONAL_PYTHON_DEPS}” ]]; then bash /scripts/docker/install_additional_dependencies.sh; fi; find /root/.local/ -name ‘*.pyc’ -print0 | xargs -0 rm -r || true ; find /root/.local/ -type d -name ‘pycache’ -print0 | xargs -0 rm -r || true’ returned a non-zero code: 1

What you expected to happen:

docker build already works in these scenarios:

• launching the above docker build using the master branch;
• using AIRFLOW_CONSTRAINTS_LOCATION="https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt" instead of AIRFLOW_CONSTRAINTS_REFERENCE on v2-0-stable branch.

How to reproduce it:

git clone https://github.com/apache/airflow.git
git checkout v2-0-stable
docker build --pull --build-arg AIRFLOW_CONSTRAINTS_REFERENCE="constraints-2.0.2" --build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" --build-arg PYTHON_MAJOR_MINOR_VERSION="3.7" --build-arg AIRFLOW_VERSION="2.0.2" --build-arg AIRFLOW_INSTALL_VERSION="==2.0.2" -t airflow:2.0.2 .

Anything else we need to know:

I’ve also noticed that https://airflow.apache.org/docs/apache-airflow/2.0.1/production-deployment.html is different from https://airflow.apache.org/docs/apache-airflow/2.0.2/production-deployment.html: there is no mention to these build args anymore.

As suggested by @ashb I’ve put set -x / set +x at the start / end of common::get_constraints_location function to have more debug info (thanks for the hint).

[Davide95]

@potiuk I’ve just tested it, it works, many thanks

[potiuk]

Shoudl be fine now if you pull the v2-0-stable branch (just merged the fix)