Add complete Kedro plugin example

Description

I’m trying to create a Kedro plugin and it took me a long time to figure out that a plugin is a separate package that exists outside of any Kedro project. I kept modifying ‘setup.py’ within my Kedro project, following the JSON example in the documentation, only to find that the kedro to_json command didn’t do anything. In retrospect it now makes sense that kedro plugins are separate python packages, but this is not obvious to new users from the documentation.

Context

New users would benefit from a more complete Kedro plugin example.

Possible Implementation

Add more detail to the Kedro plugin tutorial. Also add a complete plugin example as its own repository, similar to the kedro-examples repo.

Checklist

Include labels so that we can categorise your issue:

• Add a “Component” label to the issue
• Add a “Priority” label to the issue

[njgerner]

Our teams use the Docker functionality to schedule batch-processed runs of Kedro pipelines. Do you have an example of your workflow that you could show us?

@yetudada it’s going to depend heavily on how exactly the Kedro-Airflow plugin works. My expectation is the tool will build an Airflow DAG out of the pipelines where each node is an operator.

If that is the case then we’ll be able to throw this in to a typical CI/CD pipeline that will deploy to Airflow running on AWS.

Also, as I write this I realize building out a Kedro-Glue plugin would be useful as well. Converting Kedro pipelines to Glue jobs should be possible via CodeGenNode & related types (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-etl-script-generation.html).

That being said, thanks for the info, I’ll be on the lookout for Kedro-Airflow.

[yetudada]

@njgerner The kedro-airflow plugin should be out this week! We’re very excited about it and thank you for raising #57.

I’m going to close this issue because we used #109 to rectify this.