Circular package imports in Airflow 2.0

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): OSX
• Kernel (e.g. uname -a): Darwin Jonathons-MacBook-Pro-16-inch-2019 19.6.0 Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64 x86_64
• Install tools: Bazel ?
• Others:

What happened:

A user of bazelbuild/rules_python had trouble installing apache-airflow==2.0.0 with the Bazel build system reporting a “cycle in dependency graph” error.

Investigating the issue, I found the following in setup.py which does not seem accurate.

# Those providers are pre-installed always when airflow is installed.
# Those providers do not have dependency on airflow2.0 because that would lead to circular dependencies.
# This is not a problem for PIP but some tools (pipdeptree) show those as a warning.
PREINSTALLED_PROVIDERS = [
    'ftp',
    'http',
    'imap',
    'sqlite',
] 

Specifically this claim:

Those providers do not have dependency on airflow2.0

I appears they do have this dependency, and it does lead to circular dependencies:

.-> @py_deps//pypi__apache_airflow:pypi__apache_airflow
|   @py_deps//pypi__apache_airflow_providers_sqlite:pypi__apache_airflow_providers_sqlite
`-- @py_deps//pypi__apache_airflow:pypi__apache_airflow

In that same Github issue a colleague provides further investigation of the Airflow 2.0 wheel: https://github.com/bazelbuild/rules_python/issues/411#issuecomment-772306099

What you expected to happen:

apache-airflow==2.0.0 would not have circular deps and thus could be importable in build systems that do not allow circular dependencies (eg. Bazel)

How to reproduce it:

Issue https://github.com/bazelbuild/rules_python/issues/411 contains a reproduction repository: https://github.com/gvacaliuc/rules_python_airflow2

[thekyz]

Thanks for the links, definitely appreciated!

I managed to make it work by depending directly on the providers + apache-airflow, since then the dependency on the extras is not expressed.

For reference if somebody is encountering the same issue as me, this:

apache-airflow-providers-cncf-kubernetes==1.0.2
apache-airflow-providers-postgres==1.0.1
apache-airflow[statsd]==2.0.1

instead of this:

apache-airflow[statsd,postgres,kubernetes]==2.0.1

Thanks again @potiuk for the support!

[potiuk]

And here is explanation about airflow’s extras http://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html - it’s rather straightforward and helpful I think