Old libraries in setup.py causing dependency resolution to pull old transitive constraints (3 years+)

Dear and Wonderful Citizens,

I started to look at what libraries we have defined in the constraints-*.txt file and I am a bit surprised because we have this constraints defined on very old libraries. https://github.com/apache/airflow/blob/053afe7/constraints-3.8.txt

Update (@potiuk): -> Just for clarity: constraints are automatically generated from setup.py so this is a matter of dependencies defined there. If we are to fix it, we will have to upgrade dependencies defined in setup.py NOT the constraints themselves.

Sometimes we have defined libraries that are over 3 years old, which can cause security problems. Old versions of the library may have vulnerabilities that have probably been fixed in newer versions.

I am most concerned about dependency conflicts. Old libraries are only compatible with old libraries, which can cause problems if the user wants to use a new version of the same library.

I think it’s worth investigating where these limitations come from and why we can’t use newer versions of these libraries.

You can see the list of libraries that need updating in the Jupyter interactive notebook. https://colab.research.google.com/drive/1F5Lw8qNcxCvWaYUrGZ1x3W3v3080Dq0U#scrollTo=AfIBqzjo8UId

CC: @potiuk @ryw

[potiuk]

So it’s more than ‘month passed’ 😃 . We actually implemented actions to address it 😃

[potiuk]

I don’t think we’d want to fix deps for 1.10.* Our focus in #12636 is to make them fixed (and non-breakable in the future) for Airflow 2.0.