[user config] exception with nested structures in default context

• Cookiecutter version: 1.6.0
• Template project url: https://github.com/audreyr/cookiecutter-pypackage.git (but irrelevant in this case)
• Python version: 3.6.2 (but also irrelevant)
• Operating System: OSX (but also irrelevant)

Description:

After reading the section on user config and dictionary variables I was under the impression that defining nested YAML data structures under default_context would work (but it fails with a key error exception as is it attempting to access a key that doesn’t exist in DEFAULT_CONFIG when iterating recursively through the dictionary resulting from the user config).

After looking at the code the root cause is pretty obvious (and so is the fix) but I am wondering if this is intended behavior (still it shouldn’t fail with an exception).

Just for the sake of clarity here is a minimalist user config that makes cookiecutter fail

default_context:
  project_name: my-project
  will_fail:
    on_this: pyops

What I’ve run:

To reproduce this, copy the YAML example given above in user_config.yml and run cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git --config-file=user_config.yml The actual template is irrelevant since the failure occurs before it is even fetched. Here is the backtrace

Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/bin/cookiecutter", line 11, in <module>
    sys.exit(main())
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/cli.py", line 120, in main
    password=os.environ.get('COOKIECUTTER_REPO_PASSWORD')
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/main.py", line 54, in cookiecutter
    default_config=default_config,
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/config.py", line 109, in get_user_config
    return get_config(config_file)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/config.py", line 76, in get_config
    config_dict = merge_configs(DEFAULT_CONFIG, yaml_dict)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/config.py", line 54, in merge_configs
    new_config[k] = merge_configs(default[k], v)
  File "/usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages/cookiecutter/config.py", line 54, in merge_configs
    new_config[k] = merge_configs(default[k], v)
KeyError: 'will_fail'

Version

$ cookiecutter --version
Cookiecutter 1.6.0 from /usr/local/var/pyenv/versions/user_cfg_problem_3.6/lib/python3.6/site-packages (Python 3.6)

[alzafacon]

For those who need a workaround today, you can just overwrite the cookiecutter.json file with the values you want. and run cookiecutter --no-input <path/to/cookiecutter>

[lbrack]

It’s been a while … so I had to refresh my memory.

I think solution 1 wouldn’t work because if the user config is a subset of the entire config (why repeat config options we want to keep as default) then any nested dictionary would override whatever dictionary is at that level e.g.

default = dict(a=1, b=2, c=dict(c1=3, c2=4))
user = dict(a=1, b=4, c=dict(c1=5))
default.update(user)
# Resulting in {'a': 1, 'b': 4, 'c': {'c1': 5}}

As far as solution 2, the recursion would only happen if k is in DEFAULT_CONFIG, which, unless I am not understanding something, would not solve the problem.

Here is how I had fixed the problem. Not sure if this is the most elegant way to solve it but it does work.

def merge_configs(default, overwrite):
    """Recursively update a dict with the key/value pair of another.

    Dict values that are dictionaries themselves will be updated, whilst
    preserving existing keys.
    """
    new_config = copy.deepcopy(default)

    for k, v in overwrite.items():
        # Make sure to preserve existing items in
        # nested dicts, for example `abbreviations`
        if k not in overwrite:
            raise KeyError("option '{}' is not defined in configuration {}".format(k, overwrite))
        if isinstance(v, dict):
            new_config[k] = merge_configs(default.setdefault(k, collections.OrderedDict([])), v)
        elif isinstance(overwrite[k], list) is True and isinstance(v, list) is False:
            # The default option is a list. If the value is a scalar, we need to make sure
            # that v is in the list
            if v not in overwrite[k]:
                raise ValueError("value '{}' of option '{}' "
                                 "is not in the list of "
                                 "choices {}".format(v, k, overwrite[k]))
            new_config[k] = v
        else:
            new_config[k] = v
    return new_config