[tune] Unique trial identifier

ray version 0.7.3

I am looking for a nice solution to create and access a unique identifier (UID) inside the ray.tune.Trainable class that is restored when resuming the model. It needs to be present when calling ray.tune.Trainable._setup, also when resuming the model.

The problem I want to solve: I use wandb to monitor the model training. To restore the wandb logger, I need a UID that is accessible from within the ray.tune.Trainable class such that wandb.init(..., id=UID) appends to an existing trial. wandb is initialized and resumed in ray.tune.Trainable._setup, not in ray.tune.Trainable.resume(...).

Is there an easy way to achieve this? So far I just came up with ugly hacks (e.g. deriving the UID from config passed to ray.tune.Trainable._setup). I am sure there are other use-cases for this.

[richardliaw]

There is a unique id Trainable._experiment_id currently available but using it in _setup doesn’t quite fit your bill, as Eric mentioned.

The way I would do this right now if I didn’t want to modify the framework would be to do the initialization in either _train or _log_result, where you initialize the wandb logger only if you haven’t initialized already…

Maybe we can expose a “before_training” hook on Trainable, but that seems a bit cumbersome?

[ericl]

Why not re-initialize wandb after restore()? Currently there is no unique id available at setup time, nor does it really make sense to add it since we don’t know the id ahead of time prior to resume.