Ability to specify composition order of primary config file

There are multiple use cases for having configuration in the users config:

Use it as scaffolding and use defaults to populate it:

defalts:
  - foo: bar

foo:
   a: ???
   b: ???

In the above case foo/bar.yaml can be used to populate a and b.

An alternative approach is to allow the user to override things populated by the defaults from the config file:

defalts:
  - foo: bar

foo:
   a: 10
   b: 20

In this scenario, the user wants to force foo.a and foo.b to specific values.

Those two approaches are conflicting, each one requires a different composition order. Currently, the supported mode is the first one and the second is not possible. Composition order is roughly:

• hydra related configs
• user config file
• user provided defaults

The idea is to support explicit specification of when to merge in the users config via a special keyword in the defaults, like _self_. for example:

defalts:
  - _self_
  - foo: bar

or:

defalts:
  - foo: bar
  - _self_

In the first case, the config file would be composed before foo/bar, in the second one after.

[omry]

The primary motivation here is to address the framework use case better.

__SELF__ might be useful, but it’s already weird because hydra has it’s own defaults list that is being merged before the user defaults list, so something like:

defaults:
   - _SELF_
   - hydra/launcher: my_launcher

would actually result in hydra/launcher being merged in before SELF. There is a task to consider allowing other files to provide their own defaults (making the hydra special case a norm). those things are all interacting with one another.

There might still be room for it though, but I would want to explore it once things settles down as implementing it early might limit the design space.

by the way, a less magical form of this is to specify the file name:

my_config.yaml:

defaults:
   - my_config
   - foo: bar

[omry]

switching to target to 1.1.0 as there is a workaround for now and this is a big change with significant implications.