[tune] separate fixed configuration from searchable parameters

Describe your feature request

I’ve wrapped an existing pytorch based project with ray tune PBT. It works great 👍thank you 😃

But one thing I wish I had is, being able to input fixed configuration somewhere else other than in the configuration that is used for searching hyperparameters.

The things is, I have many options (e.g., dataset name and paths) that are completely irrelevant to the algorithm but are necessary. As far as I know, I have to include them in my hyperparameter space as list of length 1 in order to make them available to use in _setup(). But this looks messy and appears constantly when some hyperparameter perturbation occurs.

So it would be great if I could list my basic (fixed) configuration somewhere else, and use the hyperparameter space (dict) purely for tunable hyperparameters.

[crypdick]

@richardliaw would it be difficult to add support for config to (optionally) be an argparse.Namespace? Otherwise, packing args into the config dict is awkward, e.g. self.config["args"].max_epochs. It seems like it would be as simple as adding if isinstance(config, Namespace): config = vars(config).

I wrote my codebase for args, so right now I do args = config_to_args(config) in each of my functions.

def dict_to_namespace(config: Union[argparse.Namespace, Dict]) -> argparse.Namespace:
    """
    ray tournament forces us to pass args inside dict let's make args first-
    class and unpack the dict into it, with dict items taking precedence.
    """
    if isinstance(config, argparse.Namespace):
        args = config
    elif isinstance(config, dict):
        # if we do not copy first, we will have a key error
        # the next time we try to run this function
        config_ = config.copy()
        args = config_.pop("args")  # args should always be a key

        # this is the config created by Ray
        for k, v in config_.items():
            if hasattr(args, k):
                logger.warning(f"Overwriting args {k}={vars(args)[k]} with Ray Tune config {k}={v}")
            vars(args)[k] = v
    else:
        raise TypeError(f"expecting type dict or argparser.Namespace, got {type(config)}")
    return args

[richardliaw]

Are you using argparse? One option is to simply keep everything in the Namespace object and update it on the worker:

args = ArgumentParser().parse_args()
tune.run(Trainable, config={"base_args": args, "lr": ...})

BTW, this should also work:

base_config = {}

class Trainer(tune.Trainable):
    def _setup(self, config):
        self.base_config = base_config
        self.base_config.update(config)
    ...