Sweep is launching runs as individual runs instead of attaching them to sweep

• Weights and Biases version: 0.8.19
• Python version: 3.6.9
• Operating System: Linux

Description

I’m attempting to run a sweep over hyperparameters but the jobs are launching as individual runs instead of as part of the sweep. Can you describe what conditions must hold for jobs to run as part of a sweep?

What I Did

The approximate structure of my workflow is this:

train.py:

• sweep_id = wandb.sweep(sweep_config)
• wandb.agent(sweep_id, function=train_sweep(hparams, run_config, hparams_file, sweep_config))``` where train_sweep is a function which calls a model definition in model.py as below.

model.py

class Model:
wandb.init(config=config_defaults)
config = wandb.config
update model params by referring to config.param for each param
model.train()

It seems that the init in model.py is starting up a new context instead of attaching to the sweep run even though it is started with the Wandb agent, how can I fix this? Is there some way for me to “pass” the correct W&B sweep context to the model class instantiation in the other file?

Also, is there a way to instantiate a sweep without manually defining the defaults? i.e. have just it pick one from the given sweep config.

[mathemakitten]

Actually, do you have any examples of using W+B with workflows where sweeps get started up and launch a run on a new TPU? I think I’m having issues between the W+B sweep agent context and the way I’m connecting to the TPU (i.e. with TF functions like tf.config.experimental_connect_to_cluster). For some reason it works OK in single-run mode though.

[raubitsj]

I think the issue is with:

wandb.agent(sweep_id, function=train_sweep(hparams, model, train_dataset_fn, eval_dataset_fn))

function argument to wandb.agent() expects a function so unless train_sweep is returning a function you might want to do:

wandb.agent(sweep_id, function=lambda: train_sweep(hparams, model, train_dataset_fn, eval_dataset_fn))