Make config glob paths configurable by the user

Description

Is your feature request related to a problem? A clear and concise description of what the problem is: “I’m always frustrated when …”

Fix #1210

Today catalog config is loaded via the following paths (this also applies the same to parameters):

conf_catalog = conf_loader.get('catalog*', 'catalog*/**')

https://github.com/kedro-org/kedro/blob/e78990c6b606a27830f0d502afa0f639c0830950/kedro/framework/context/context.py#L356

These are hardcoded in the library and the user of a project cannot change without getting creative. I think there would be a lot of user value (1) making these configurable (2) being transparent with how this works.

Context

Why is this change important to you? How would you use it? How can it benefit other users?

• It feels restrictive to stop users controlling this when so much of the framework is configurable
• We have evidence of users want to do _catalog suffixes (as opposed to prefixes) which aren’t supported by the default pattern

Possible Implementation

(Optional) Suggest an idea for implementing the addition or change.

• In 0.18.x it may make sense to expose in settings.py

[AntonyMilneQB]

I’m going to reopen this because I think it was closed on the misunderstanding that this is already possible through a custom ConfigLoader, which as far as I can tell it’s not. It’s really about a custom KedroContext.

This is a request I’ve heard several times before (or at least the question of “what are the patterns kedro looks for in finding catalog/params”), and I think it makes a lot of sense to make this more transparent and configurable.

I’m not sure exactly how far we should go into making these patterns exposed to the user (e.g. whether we should make them configurable in settings.py, whether CONTEXT_ARGS should be configurable), but given that CONTEXT_CLASS is already exposed in settings.py it would be very easy to make these patterns configurable simply by making them class properties/methods than can be directly overridden rather than buried inside other methods.

This won’t immediately meet @DebanjanBanerjeeQB’s ask that we shouldn’t need to make a custom class just for a small change to naming convention, since you would still need to make a custom KedroContext class to change the patterns. But at least it would make configuring these patterns possible. After that we can consider whether users should be able to change them without needing to make a custom KedroContext.

[datajoely]

So I actually disagree here - I think it’s highly likely that a novice user would like to control this without the added complexity of the creating a custom class.

The smallest version of this I can imagine involves inheriting from the config loader you were already using, and overriding _get_config_credentials(self) and the params(self) which is type @property. I really think this is prohibitively complex for novices.

What about adding the defaults as constructor arguments?