Recursive defaults support

Hydra does not currently support a recursive defaults list. the defaults are only specified at the primary config file. There is some special support for Hydra itself, but this can be ignored for the purpose of this issue.

Recursive defaults list are tricky, it’s important to ensure intuitive and consistent order of merging.

Proposed logic

if the defaults in the primary file are in that order: main.yaml

defaults:
  - aa: x1
  - bb: x2

The relative order should be preserved. it is however possible for other files to be injected in between aa and bb, for example if aa/x1 had it’s own defaults:

aa/x1.yaml

defaults:
  - cc: x3
  - dd: x4

the resulting defaults list might looks like:

  - aa: x1
  - cc: x3  # from aa/x1 
  - dd: x4  # from aa/x1 
  - bb: x2

Let’s look at a more complicated case:

main.yaml

defaults:
  - aa: x1
  - bb: x2
  - cc: must_win

aa/x1.yaml

defaults:
  - cc: x3
  - dd: x4

In this case, the result should be: the resulting intermediate list will looks like:

  - aa: x1
  - cc: x3
  - dd: x4
  - bb: x2
  - cc: must_win

but the final result will be:

  - aa: x1
  - cc: must_win
  - dd: x4
  - bb: x2

Note how cc retains it’s order in the list, but got the last value.

The rule here is:

1. Composition order determined by the first time a configuration group is introduced (the db in db=mysql).
2. Composition choice (the mysql in db=mysql) is determined by the last mention of the group).

[omry]

The plan is to implement this for 1.1.

[omry]

This feature (or more accurately the one from #1170 a complete rewrite that is not merged yet) is more or less ready (mostly only documentation and some followups are missing).

There are a number of other things which are also planned for Hydra 1.1. You can follow the milestone for 1.1 here.

I am guestimating that initial release candidate will be ready sometimes during H1 2021.

Keep in mind that Hydra 1.1 will take even longer to hit fbcode due to the monorepo nature of it.