question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Refactor unit tests that depend on deprecated airflow example pipeline components

See original GitHub issue

Is your feature request related to a problem? Please describe. The main purpose of the Airflow component examples connector was to provide “ready-for-exploration” pipeline components:

The component files marked above with ‘(core)’ will be available via the Airflow package connector after https://github.com/elyra-ai/elyra/pull/2409 is published. Likewise, component files marked above with ‘(community provider)’ will be available via the Airflow provider package connector after https://github.com/elyra-ai/elyra/pull/2416 is published.

Once those PRs were published, the existing Airflow component examples connector no longer provides any value as is.

Describe the solution you’d like

The Airflow component examples connector should be deprecated, withdrawn, or re-purposed.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
kiersten-stokescommented, Feb 9, 2022

We’ll also want to either remove or find some other way to test our integration tests for custom components, namely:

  • 'airflow pipeline should display custom components'
  • 'kfp pipeline should display custom components'

Certain backend tests also rely on the examples catalog, so we’ll have to take a look at that too

0reactions
kiersten-stokescommented, Feb 10, 2022

None of the tests actually run anything on an Airflow cluster, right? If that’s the case it’s probably sufficient to create test-only operators, which could be no-ops

Correct – yeah I forget the exact extent of how operators are used in each test, but I don’t think it would be too difficult to replace them with others

Read more comments on GitHub >

github_iconTop Results From Across the Web

Release Notes — Airflow Documentation
operators , airflow.sensors packages are now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer ......
Read more >
Changelog — great_expectations documentation
[BUGFIX] Revert refactor to Datasource instantiation logic in DataContext (#6535) ... [MAINTENANCE] Expectation suite init unit tests + types (#5957).
Read more >
ensuring your DAGs work before going into production
Testing Airflow workflows - ensuring your DAGs work before going into production. Watch later. Share. Copy link.
Read more >
1.1.7 (core) / 0.17.7 (libraries) - Dagster Docs
New documentation for dagster-airflow on how to start writing dagster code from an airflow background. 1.1.6 (core) / 0.17.6 (libraries)#. New#. [dagit] ...
Read more >
Unit Testing Your Airflow Data Pipeline - Level Up Coding
Unit tests allow us to ensure that our DAG task has been properly loaded and that its structure is coherent. A more advanced...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found