Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Enhancement] Multiple model iterations per Optuna trial and mean performance objective

See original GitHub issue

I currently have the problem that, a lot of times, the results Optuna optimization produces are not really too optimal, due to the stochastic nature of RL training. For example, training 3 agents with the same set of hyperparameters can result in 3 completely different learning curves (at least for the environment I’m training on). Might it make sense to implement the optimization code in way, such that for each trial multiple agents are trained, and the mean or median performance is reported to Optuna instead?

Inside utils/ hyperparameter_optimization, line 713, I saw your comment “# TODO: eval each hyperparams several times to account for noisy evaluation”. Is that maybe exactly what you mention there?

I already had a look at the code and thought a little bit about how one might be able to do that. If somebody would be interested I could implement it and issue a pull request!

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:1
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

qgallouedeccommented, Mar 30, 2022

Please do =)

Let’s open a draft PR and continue the discussion there.

araffincommented, Mar 30, 2022

If you open a PR, I would be happy to contribute.

Please do =)

By training multiple models simultaneously. Something like

I was afraid of that answer… yes it does work but not for image-based environment and requires beefy machine anyway (for instance for DQN on Atari, a single model may require 40GB of RAM). We also need to check if the model.learn(reset_num_timesteps=False) works well with schedules.

50 or so models simultaneously, without having memory problems or anything.

I would run only maximum 3-5 models simultaneously, unless the env is very simple and the network small.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Optuna Guide: How to Monitor Hyper-Parameter Optimization ...
We define a function, in this case objective, which takes an object called trial. This trial object is used to construct a model...
Read more >
optuna.trial.Trial — Optuna 3.0.4 documentation
A trial is a process of evaluating an objective function. This object is passed to an objective function and provides interfaces to get...
Read more >
Optuna: Simple Guide to Hyperparameters Tuning / Optimization
It'll execute the objective function more than once by giving different Trial instances each having different hyperparameters combinations.
Read more >
State-of-the-Art Machine Learning Hyperparameter ...
Optuna enables users to adopt state-of-the-art algorithms for sampling hyperparameters and pruning unpromising trials. This helps to speed up optimization ...
Read more >
optuna/optuna - Gitter
Is there a rule of thumb for how to choose the number of epochs per trial? ... I'm trying to use multi-objective optimization...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found