override `runner.run` line in KedroContext run method

Description

This is for only_missing option without kedro team support. I searched issues and found Kedro doesn’t support this option. (#30, #60, #82 )

https://github.com/quantumblacklabs/kedro/issues/30#issuecomment-513215607

the Kedro team does not intend to support this as core functionality. It can, however, now be more easily added in the project’s capabilities, thanks to the addition of KedroContext!

so I did it on my own risk by defining my ProjectContext class, Thanks! but there is something troublesome point.

Context

When I try to add the only_missing option, I must copy the entire KedroContext’s run method and paste it and add if only_missing .... but it’s something dirty way and I have to do it again when I update the kedro version.

If the line run_result = runner.run(filtered_pipeline, catalog, run_id) move to another method, I can override this.

I don’t come up with another situation that wants to modify this line, but it’s extensible and Kedro Team doesn’t have to support the only_missing option.

Possible Implementation

https://github.com/quantumblacklabs/kedro/blob/0.16.5/kedro/framework/context/context.py#L759 just move this line to KedroContext’s method, and users can modify it.

Possible Alternatives

Or, if Kedro hooks can do side-effect that effects filtered_pipeline and so on, I can also implement same feature as Kedro hooks. For example, on before_pipeline_run I may be able to do it by modifying the pipeline with the same way in the run_only_missing method.

[WaylonWalker]

I will echo here that maintaining modified project templates can be tiresome as the framework changes. I do appreciate that project templates are getting better and more simple with each change.

[lorenabalan]

HI @miyamonz apologies for the late reply. For full transparency, we’re actually planning on re-evaluating the whole Runner abstraction in the future, as we don’t believe it’s suitable anymore in its current form, if at all. The vision is that KedroSession would basically be, conceptually, the combination of Journal and Runner. Because of that, and given there are current workarounds available to make this doable (e.g. using a custom runner or a custom CLI flag), we won’t be prioritising this change. I appreciate this is not what you’d like to hear, but hope it makes sense.