Refactor unit tests that depend on deprecated airflow example pipeline components
See original GitHub issueIs your feature request related to a problem? Please describe. The main purpose of the Airflow component examples connector was to provide “ready-for-exploration” pipeline components:
- bash_operator.py (core)
- email_operator.py (core)
- http_operator.py (
apache-airflow-providers-http
community provider) - slack_operator.py (
apache-airflow-providers-slack
community provider) - spark_sql_operator.py (
airflow.providers.apache.spark
community provider) - spark_submit_operator.py (
airflow.providers.apache.spark
community provider)
The component files marked above with ‘(core)
’ will be available via the Airflow package connector after https://github.com/elyra-ai/elyra/pull/2409 is published. Likewise, component files marked above with ‘(community provider)
’ will be available via the Airflow provider package connector after https://github.com/elyra-ai/elyra/pull/2416 is published.
Once those PRs were published, the existing Airflow component examples connector no longer provides any value as is.
Describe the solution you’d like
The Airflow component examples connector should be deprecated, withdrawn, or re-purposed.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:5 (5 by maintainers)
Top GitHub Comments
We’ll also want to either remove or find some other way to test our integration tests for custom components, namely:
'airflow pipeline should display custom components'
'kfp pipeline should display custom components'
Certain backend tests also rely on the examples catalog, so we’ll have to take a look at that too
Correct – yeah I forget the exact extent of how operators are used in each test, but I don’t think it would be too difficult to replace them with others