question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Doubt about combining AsyncHyperBand with HyperOpt

See original GitHub issue

HyperOptSearch only passes the results to hyperopt when a trial is completed: https://github.com/ray-project/ray/blob/16e9dfd2e197995f853e4f3a504edbe703e43093/python/ray/tune/suggest/hyperopt.py#L183-L193

However, when we use AsyncHyperBand to schedule trials, a lot of trials are early terminated and their results are useless for hyperopt.

Do you have any practical advice to alleviate this problem?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
richardliawcommented, Nov 3, 2019

OK this is great! Thanks for doing this. I’ll hopefully merge in the upcoming week.

On Sat, Nov 2, 2019 at 7:58 PM lanlin notifications@github.com wrote:

@richardliaw https://github.com/richardliaw I verified that using early terminated trials could achieve better search results in my case.

In addition, for using or not using, I used warn-up records by changing this line https://github.com/ray-project/ray/blob/1a3e97cf232fe53984ca998d87ee0f37769671a7/python/ray/tune/schedulers/async_hyperband.py#L146 to if not recorded or len(recorded) < 50:.

I did not compare the performance of the using and the not using in the absence of the warm-up records.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/6056?email_source=notifications&email_token=ABCRZZJH3PDXCL22DI2DYPTQRY44JA5CNFSM4JHLRXH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC5JXDI#issuecomment-549100429, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCRZZLPEG64LZQNZH6NH23QRY44JANCNFSM4JHLRXHQ .

0reactions
richardliawcommented, Mar 26, 2020

Closing this for now; but please bring up any other suggestions for how we can improve this.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Ray Tune ; combine population based training schedule with ...
Are Population Based Training (PBT) and HyperOpt Search combinable ? The AsyncHyperBandScheduler is used in the Hyperopt Example of ray.tune.
Read more >
A Novice's Guide to Hyperparameter Optimization at Scale |
A Novice's Guide to Hyperparameter Optimization at Scale · Async Successive Halving Algorithm (ASHA — scheduler) · Async Hyperband (AHB — ...
Read more >
Hyperopt - Ludwig
combined is a special output feature that allows to optimize for the aggregated loss and metrics of all output features. metric is the...
Read more >
Trial Schedulers (tune.schedulers) — Ray 2.2.0
In Tune, some hyperparameter optimization algorithms are written as “scheduling algorithms”. These Trial Schedulers can early terminate bad trials, pause trials ...
Read more >
Part 2: Hyperopt. - Towards Data Science
We will talk about hyperopt hyperparameter optimization library. ... we can see that by combining hp.choice with other sampling methods we ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found