CI: convert tests to use Breeze tests command

The tests that are run in CI should use breeze and parallellism implemented in Python.

NOTE: We should pay an attention to provide better error feedback when docker containers could not be started - see https://github.com/apache/airflow/issues/23523

[potiuk]

Could you point me to any one of the tests in ci.yml file that has to be replaced with Breeze @potiuk ?

Sure. Any test that is currently run by “ci_run_airflow_testing.sh”:

https://github.com/apache/airflow/blob/main/.github/workflows/ci.yml#L1061

[potiuk]

Yeah. It’s an ambitious task though 😃.

Just pay attention to the --parallell flag I’ve implemented in other commands (pull-images for one). What we’ve been doing so far - we run the tests in parallel for different test types when we had enough CPUs. But this was overly complex with gnu-parallel and we should do it as single breeze command with --parallel flag as parallelism is much better handled in Python and easier to reason about.

One of the goals of the task is not only to duplicate the current behaviour, but also improve it - using python we might want to do a bit better when it comes to grabbing the output of such parallel tests and displaying them to the user. The current solution “looks” good, but there were a few things that were just too complex to implement in Bash easily, but are entirely possible if we do it in Python.

There are a few properties for the target solution:

• we need to see the progress of parallel tests (probably slightly different than the current approach as we can be a bit smarter in Python - from each parallel task we can see the progress of tests in % and just display the progress for each parallel task rather than what we did so far (showing last few lines from every parallell test type run)
• We need to capture the output from each such parallel run and when all of them are complete we need to print them in separate folded CI group - marking the test types that failed in “red” and those that succeeded with “green”. You can see at the current output of tests how it looks like it’s pretty nice now.
• Ideally we should also print a nice summary of only failed stuff after that so that the user sees immediately what failed without having to open the folded group (this shoudl be used if you want to see more details)
• when there is an error we should also (this is currently done) grab the logs of containers and save them so that they can be uploaded as artifacts (this enormosly helps to diagnose problems).
• also as mentioned in #23523 - when we fail to start docker-compose at all (for example when containers are unhealthy) we should make a better visual indication what’s the actual error (marked in red)

The difficult thing is that it has to be “all in one go” - so it will be quite a big PR overall.

Also pay attention to a change I am about to submit - I am doing a refactor of packages/modules in Breeze - so you might want to wait with some heavy implementation once it is merged.