Allow extra hooks to be passed via the Kedro CLI

Description

Is your feature request related to a problem? A clear and concise description of what the problem is: “I’m always frustrated when …”

I should be able to run with or without a set of optional hooks. For example:

kedro run --hooks src.hookshot.hooks.TeePlugin

or

kedro run --hooks src.hookshot.hooks.TeePlugin,src.hookshot.hooks.CachePlugin

Context

Why is this change important to you? How would you use it? How can it benefit other users?

Not all hooks are core to functionality. Using something like a debugging hook shouldn’t require a code change.

This is also helpful for testing different hooks as part of a CI/CD process.

Possible Implementation

See https://github.com/deepyaman/hookshot/commit/0f792f9112031fc2642d30e161c21a0b0c9f539d#diff-ebf803a458716ccda133fadc42e45057 (would add it to KedroContext though).

[limdauto]

@deepyaman your timing is uncanny. We are building this as we speak 😂

However, we are going with an opt-out model, hence @yetudada’s question. Basically, after user installs a plugin, we will automatically register the plugin’s hooks through a dedicated kedro.hooks entrypoint. This is similar to how global_commands and project_commands are working at the moment. This is the pytest’s model. We will also provide a mechanism for users to disable hooks that they don’t want to run on a per-run basis.

I’m curious what you and @WaylonWalker think about these two models. Do you think an opt-out (auto-discovery) model or an opt-in model (your proposal) would work better?

[lorenabalan]

Hi @deepyaman thank you for sharing, these are some really interesting questions and comments. I’m adding some of my thoughts, let me know what you make of them.

It feels like the opt-in / opt-out choice really just depends on how easy it is to opt out. Indeed uninstalling stuff would be a hassle. I saw pytest uses an env variable to blanket-disable auto-discovery, which we considered for Kedro. But they also have another environment variable to specify which plugins to use (or just a variable), so maybe something similar can be mirrored in Kedro (it’d be different conf environments for different plugin combinations, so for your benchmarking example you’d have 3 environments). Hooks behaviour is sth that feels too heavy to just be configured at runtime, because it strongly interacts with or change the run flow, to me it’d be safer as static config.

There’s a model in my head that makes me quite reticent about feeding hooks through the CLI, where it’s not a developer that messes with the job spec, so the less technical it is, or the less code knowledge you need to understand it, the better. The person on support doesn’t need to understand what hooks to activate, they can just switch to a different environment and job done, if the env is there. If not, a developer should create it, which feels intentional.