Support parallel sweeps with local controller

wandb --version && python --version && uname

• Weights and Biases version: 0.8.18
• Python version: 3.7.4
• Operating System: Linux-4.4.0-1090-aws-x86_64-with-debian-stretch-sid

Description

I was trying to run multiple sweeps in parallel using a local controller. The documentation isn’t very clear, so I thought it was supported, with the only caveat that the controller would only schedule a new run once the scheduled queue was empty.

Based on the comments from wandb_controller I was surprised that only one of my workers were running and all others were idle. Maybe the definition of “scheduled” leaves room for confusion?

    Protocols:
        Scheduling a run:
        - client controller adds a schedule entry on the controller.schedule list
        - cloud backend notices the new entry and creates a run with the parameters
        - cloud backend adds a scheduled entry on the scheduler.scheduled list
        - client controller notices that the run has been scheduled and removes it from
          controller.schedule list
    Current implementation details:
        - Runs are only schedule if there are no other runs scheduled.

What I Did

I defined the parameters for my local sweep in a YAML file:

program: run.py
method: bayes
metric:
  name: mape
  goal: minimize
controller:
  type: local
parameters:
  days:
    min: 1
    max: 7
  smoothing:
    min: 0.0
    max: 0.995

and ran from the command line:

$ wandb sweep --controller --verbose sweep.yaml

PS: The problem goes away when you run the cloud version of the sweep. I understand this may not be an urgent feature, but it would be great to at least make it clear that local sweeps do not support parallel agents for now.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label enhancement to this issue, with a confidence of 0.72. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[Chen-Cai-OSU]

I also need this feature to run several jobs (each one has low gpu utilization) within one sweep. It will be a very nice feature if wandb can include it.