Latest SQLAlchemy (1.4) Incompatible with latest sqlalchemy_utils

Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Mac OS Big Sur
• Kernel (e.g. uname -a):
• Install tools: pip 20.1.1
• Others:

What happened:

Our CI environment broke due to the release of SQLAlchemy 1.4, which is incompatible with the latest version of sqlalchemy-utils. (Related issue)

Partial stacktrace:

  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
    from flask_appbuilder.models.sqla.interface import SQLAInterface
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
    from sqlalchemy_utils.types.uuid import UUIDType
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
    from .aggregates import aggregated  # noqa
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
    from .functions.orm import get_column_key
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
    from .database import (  # noqa
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
    from .orm import quote
  File "/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
    from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/Users/samwheating/Desktop/tmp_venv/lib/python3.7/site-packages/sqlalchemy/orm/query.py)

I’m not sure what the typical procedure is in the case of breaking changes to dependencies, but seeing as there’s an upcoming release I thought it might be worth pinning sqlalchemy to 1.3.x? (Or pin the version of sqlalchemy-utils to a compatible version if one is released before Airflow 2.0.2)

What you expected to happen:

airflow db init to run successfully.

How to reproduce it:

1. Create a new virtualenv
2. pip install apache-airflow
3. airflow db init

[kaxil]

@jonasrla If you install Airflow with constraints file you won’t need to do that:

https://github.com/apache/airflow#installing-from-pypi

Example:

pip install apache-airflow==2.0.1 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt"

[jonasrla]

Hello,

First of all, to see how things are working out about this issue checkout #https://github.com/kvesteri/sqlalchemy-utils/issues/505

I worked around this issue by forcing pip install SQLAlchemy==1.3.23 after installing airflow, so it overrides the newer SQLAlchemy version.