Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[tune] How to save the ax and ray.tune search progress for the next training session?

See original GitHub issue

What is your question?

I’m using tune.Trainable and tune.run to search hyparams, like https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/mnist_pytorch_trainable.py.

I want to save the hyparam search progress for another training session since my search space is so large and I don’t want to waste any experiments.

And here are my questions:

What do I need to save? ax.client is a must, but what about ray.tune part? Maybe the scheduler?
How to save ax.client and ray.tune related things using tune.run or through other mechanism?

Thanks!

Ray version and other system information (Python version, TensorFlow version, OS): Ray 0.8.4, Python 3.8.2, Pytorch 1.5.0, arch linux

Issue Analytics

State:
Created 3 years ago
Comments:8 (3 by maintainers)

Top GitHub Comments

1reaction

richardliawcommented, Nov 19, 2020

I think this is now fixed on master.

0reactions

xiangdalcommented, Oct 5, 2021

sure!

Top Results From Across the Web

Training (tune.Trainable, session.report) — Ray 2.2.0

You can save and load checkpoints in Ray Tune in the following manner: from ray import tune from ray.air import session from ray.air.checkpoint...

Ray Tune FAQ — Ray 2.2.0 - the Ray documentation

Ray Tune offers many different search algorithms and schedulers. ... If you only call session.report() once at the end of the training, the...

Search Algorithms (tune.search) — Ray 2.2.0

Certain search algorithms have save/restore implemented, allowing reuse of learnings across multiple tuning runs. search_alg = HyperOptSearch() tuner_1 = tune.

Getting Started — Ray 2.2.0

Getting Started#. This tutorial will walk you through the process of setting up a Tune experiment. We'll start with a PyTorch model and...

Console Output (Reporters) — Ray 2.2.0

Setup progress reporter for a new Ray Tune run. This function is used to initialize parameters that are set on runtime. It will...