[tune] Hyperband Scheduler Not working

What is the problem?

Hyperband scheduler doesn’t work. Some experiments succeed but about 50% fail. I tried Ray version 0.7.2 and 0.9.0.dev0 running python 3.7 on Ubuntu 18.04.

The def update_trial_stats(self, trial, result) function fails and here is the error:

Failure # 1 (occurred at 2020-02-02_14-45-58)
Traceback (most recent call last):
  File "/home/kaleab/anaconda3/envs/research/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 461, in _process_trial
    self, trial, flat_result)
  File "/home/kaleab/anaconda3/envs/research/lib/python3.7/site-packages/ray/tune/schedulers/hyperband.py", line 172, in on_trial_result
    bracket.update_trial_stats(trial, result)
  File "/home/kaleab/anaconda3/envs/research/lib/python3.7/site-packages/ray/tune/schedulers/hyperband.py", line 382, in update_trial_stats
    assert delta >= 0
AssertionError

I have run validate_save_restore(trainable_class, config=validate_save_config) and validate_save_restore(trainable_class, config=validate_save_config, use_object_store=True) and they both succeed.

Reproduction

Run hyperband on CIFAR 10, with the following config:

sched = HyperBandScheduler(
               time_attr="training_iteration",
               metric="accuracy",
               mode="max",
               max_t=ray_config['epochs'])

Trainable class similiar to https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch_trainable.py, with the exact same save and restore methods.

[richardliaw]

@semin-park currently investigating this!

[semin-park]

BOHB is not working either due to the same reason.