Nested parameters update via command line / config file

Description

Currently, when passing a parameter to override configurations like kedro run --params key:value, it only supports top-level key but not nested dictionary. This is not flexible for experiments.

Similarly, kedro run --config=config.yml, overwriting via config file suffers from the same top-level key only problem.

Ideally.

# paramters.yml
group1:
   key1: 1
   key2: 2
   key3: 3

Ideally, this should be the expected result

# paramters.yml
group1:
   key1: 1
   key2: 2
   key3: 4

Current behavior

# paramters.yml
group1:
   key3: 4

Related #605

Context

A lot of parameters are organized in groups instead of being a top-level key.

Possible Implementation

Maybe a simple nest dict update is enough, I am not sure

Possible Alternatives

(Optional) Describe any alternative solutions or features you’ve considered.

[AntonyMilneQB]

Hello @noklam, I totally understand the need for this and agree that there should be a better mechanism for overwriting a nested parameter. But I’m not sure how easy it is to resolve just this without some more holistic solution that reworks how configuration is handled more generally, and as per @lorenabalan’s comment in #605 there’s a bit of a conversation ongoing about this (N.B. @datajoely).

If we allow kedro run --params key.subkey:value to overwrite a nested parameter then it immediately raises the question of why we don’t allow this sort of nested overwrite in configuration environments as you wanted to do in #605. So while it sounds like a very useful feature to add, it leads to inconsistency between the CLI and the parameters.yml approaches to specifying configuration. In an ideal world you would be able to do this sort of thing from both CLI and parameters.yml, but that is part of the larger conversation I think (and in addition to overwriting keys should also consider things like deleting and adding keys).

This nested overwriting behaviour is already possible through use of TemplatedConfigLoader and globals. I know that’s a bit hacky and not a nice solution, but at least it is possible to currently do this. So my opinion would tend to be: very nice idea, I 100% agree we need a better solution than we currently have, but I think it needs to come as part of the complete solution rather than just implementing part of it that only works using the CLI.

[noklam]

#927 is an equivalent PR that get merged