Failure during "airflow db init" on fresh airflow 2.0 installation

Apache Airflow version: 2.0

Environment:

• Cloud provider or hardware configuration: Local development environment using docker
• OS (e.g. from /etc/os-release): macOS Catalina (10.15.7)
• Kernel (e.g. uname -a): Darwin
• Install tools: poetry
• Others: docker 20.10.0, postgres, Python 3.7.9

What happened:

A fresh installation of airflow 2.0 seems to be failing on airflow db init with what looks like a 3rd-party library exception (see traceback below). I searched for related issues on github/google; but didn’t find anything useful.

What you expected to happen:

Given I couldn’t find any useful information online, this makes me think the problem is my environment. I will continue to look at it, but in the meantime I’d like to put this out there in case anyone else has had a similar issue.

How to reproduce it:

1. Install poetry package manager

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

2. Create a docker-compose.yml file inside a new directory

version: "3.9"
services:
  db:
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
      POSTGRES_PORT: 5432
    image: postgres
    ports:
      - 5432:5432

3. Create a pyproject.toml file inside the same directory as (2)

[tool.poetry]
name = "airflow-2-docker-example"
version = "0.1.0"
description = ""
authors = ["name <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.7"
apache-airflow = "^2.0.0"
psycopg2-binary = "^2.8.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

4. Install python packages and bring up the database

poetry install
docker-compose up -d

5. Initialize airflow database

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost:5432/airflow poetry run airflow db init

Anything else we need to know:

Traceback

hankehly ~/src/airflow-2-docker-example $ AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost:5432/airflow poetry run airflow db init
DB: postgresql+psycopg2://airflow:***@localhost:5432/airflow
[2020-12-18 19:23:20,344] {db.py:678} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 849da589634d -> 2c6edca13270, Resource based permissions.
Traceback (most recent call last):
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
    db.initdb()
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
    upgradedb()
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/utils/db.py", line 688, in upgradedb
    command.upgrade(config, 'heads')
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 522, in run_migrations
    step.migration_fn(**kw)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 310, in upgrade
    remap_permissions()
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 287, in remap_permissions
    appbuilder = create_app(config={'FAB_UPDATE_PERMS': False}).appbuilder
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/airflow/www/app.py", line 74, in create_app
    flask_app.config.from_pyfile(settings.WEBSERVER_CONFIG, silent=True)
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/flask/config.py", line 132, in from_pyfile
    exec(compile(config_file.read(), filename, "exec"), d.__dict__)
  File "/Users/hankehly/airflow/webserver_config.py", line 21, in <module>
    from flask_appbuilder.security.manager import AUTH_DB
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/flask_appbuilder/security/manager.py", line 13, in <module>
    from flask_openid import OpenID
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/flask_openid.py", line 26, in <module>
    from openid.store.filestore import FileOpenIDStore
  File "/Users/hankehly/src/airflow-2-docker-example/.venv/lib/python3.7/site-packages/openid/__init__.py", line 52, in <module>
    if len(version_info) != 3:
TypeError: object of type 'map' has no len()

[petobens]

I had to fork flask-openid to make it work:

[tool.poetry]
name = "airflow-new"
version = "0.1.0"
description = ""
authors = ["petobens <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.8"
flask-openid = {git = "https://github.com/petobens/flask-openid.git"}
apache-airflow = "^2.0.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

basically removing the sys.version_info check that made poetry install the Python2 dependency instead of the correct Python3 one. Since Airflow 2.0 only supports Python >=3.6 can Airflow either use my fork or a new flask-openid fork with this workaround? (the official flask-openid library hasn’t been updated in almost 5 years)

[Limess]

Not to ignore the above discussion, but for the benefit of anyone else using poetry on top of airflow in a workflow similar to the following who is experiencing issues with flask-openid:

1. Use the official Airflow docker image
2. Export requirements from poetry so we can use the Airflow constraints file
3. Globally install on top of the base Airflow dependencies using the airflow constraints file to ensure we get what the Airflow team deems ‘golden’

I had to add python3-openid to our poetry.toml as above, but I also had to remove python-openid from the output requirements.txt, otherwise we still had it floating around and causing issues. As the poetry issues (https://github.com/python-poetry/poetry/issues/1287) says, this is a problem with flask-openid, not poetry :

RUN export PYTHON_MAJOR_MINOR_VERSION=$(python -c 'import sys; print("%s.%s"% (sys.version_info.major, sys.version_info.minor))') \
  && AIRFLOW_MINOR_VERSION=$(echo "$AIRFLOW_VERSION" | cut -d "." -f 1)-$(echo "$AIRFLOW_VERSION" | cut -d "." -f 2) \
  && curl -sSL "https://raw.githubusercontent.com/apache/airflow/constraints-$AIRFLOW_MINOR_VERSION/constraints-$PYTHON_MAJOR_MINOR_VERSION.txt" > ./airflow-constraints.txt \
  && poetry export --without-hashes -f requirements.txt -o ./requirements.txt \
  # flask-openid does not correctly specify version constraints https://github.com/python-poetry/poetry/issues/1287
  && echo "remove python-openid from poetry packages as it's pulled in incorrectly by flask-openid" \
  && sed -i '/^python-openid==/d' ./requirements.txt \
  && pip install --user --no-cache-dir --upgrade pip==${PIP_VERSION} \
  && pip install --user --no-cache-dir --no-warn-script-location -r ./requirements.txt --constraint ./airflow-constraints.txt \
  && rm -rf ~/.cache ./requirements.txt ./airflow-constraints.txt