Sweeps are unable to change values from config-defaults.yaml with allow_val_change=True in wandb.init()

Describe the bug Unable to start the sweep when using local config file config-defaults.yaml containing some default parameters.

My issue is related to this one #1159. With default parameters as dictionary passed to it works well wandb.init(), but when I use local config file in the root of the project config-defaults.yaml and the option allow_val_change=True in wandb.init() I get the following error:

Problem at: train.py 13 <module>
Traceback (most recent call last):
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_init.py", line 575, in init
    run = wi.init()
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_init.py", line 379, in init
    run = Run(config=config, settings=s)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 251, in __init__
    self._config._update(config)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 152, in _update
    sanitized = self._sanitize_dict(parsed_dict, allow_val_change)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 198, in _sanitize_dict
    k, v = self._sanitize(k, v, allow_val_change)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 218, in _sanitize
    ).format(key, self._items[key], val)
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "filter_size" from 0.5 to 6
If you really want to do this, pass allow_val_change=True to config.update()
wandb: ERROR Abnormal program exit
Traceback (most recent call last):
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_init.py", line 575, in init
    run = wi.init()
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_init.py", line 379, in init
    run = Run(config=config, settings=s)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_run.py", line 251, in __init__
    self._config._update(config)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 152, in _update
    sanitized = self._sanitize_dict(parsed_dict, allow_val_change)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 198, in _sanitize_dict
    k, v = self._sanitize(k, v, allow_val_change)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_config.py", line 218, in _sanitize
    ).format(key, self._items[key], val)
wandb.sdk.lib.config_util.ConfigError: Attempted to change value of key "filter_size" from 0.5 to 6
If you really want to do this, pass allow_val_change=True to config.update()

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 13, in <module>
    wandb.init(project="xxx", entity="xxx", allow_val_change=True,)
  File "/path/to/env/lib/python3.6/site-packages/wandb/sdk/wandb_init.py", line 612, in init
    six.raise_from(Exception("problem"), error_seen)
  File "<string>", line 3, in raise_from
Exception: problem

To Reproduce use a modified minimum example from: #1159.

Train.py

import wandb

# defaults = dict(seed=42, filter_size=3)
# wandb.init(entity="xxx", allow_val_change=True, project="xxx",config=defaults)

wandb.init(entity="xxx", allow_val_change=True, project="xxx")
print('\n\nHyperparams:')
print(wandb.config)
print('\n\n')
log = 2*wandb.config["filter_size"] - 0.01*wandb.config["seed"]
wandb.log({"output":log})
wandb.finish()

Sweep.yml

project: xxx
entity: xxx
program: train.py
method: bayes
metric:
  goal: minimize
  name: output
parameters:
  seed:
    distribution: int_uniform
    max: 100
    min: 42
  filter_size:
    distribution: int_uniform
    max: 7
    min: 3

config-defaults.yml

seed:
  value: 42
filter_size:
  value: 0.5

Steps to reproduce the behavior:

1. activate your env
2. wandb sweep sweep.yaml
3. start an agent

Versions:

• Ubuntu 20.04
• wandb0.10.12 and wandb0.10.14
• python3.6.9

[datduong]

Was this fix released in the version 0.12.10? I am seeing the same issue in my code. Thanks.

[aidanjdonohue]

@datduong bumping this thread