Augmenting trial run metadata with metadata from `Runner.stop` call

Hi, I would like to thank you for such a detailed abstraction of experimentation steps. Although it needs a very deep dig of the modules until understanding the purpose of a component (sometimes I misinterpret too), eventually, all pieces fit beautifully.

I want to discuss a usage case that relies on the implementation of a Runner. I checked that the only required method to implement is Runner.run(), which may return some metadata about deployment. I assume that I can use this method to get the arms of the trial and prepare the actual parameters to configure and start the actual process. Then, possibly it collects some other metadata during running, which will later be used to compute the cost of a metric using Metric.fetch_trial_data(). This part may not be as I predicted but, unlike synthetic metrics that implement an offline function f(), I couldn’t think of another way to handle process data.

In a specific case, I want to use Runner.run() to open a TCP connection on a real-time target computer and deploy the required parameters. However, the target cannot transfer real-time process data before stopping the process because it would overload the computer. So, Runner.run() will not return any process metadata. Conceptually, I would use Runner.stop() to stop the process, close the connection and collect the data that my metric will need to compute its cost. However, BasicTrial.complete() does not collect any metadata.

As a work-around, I would implement the Runner.run() to execute all the above processes, but then, what would be the purpose of Runner.stop() because eventually, I will define something like stop_my_process() and call it in Runner.run().

Probably, I will work this out by implementing a MyTrial that overrides only .complete() such that:

...
metadata = not_none(self._runner).stop(self)
self.update_run_metadata(metadata) if metadata else pass
...

But I would like to get your comments about it: would it be considered directly in BasicTrial in the future, or is there any other way to do it more ax-like?

[Balandat]

@ugurmengilli I just realized that I happened to work on exactly this feature without actually being aware of this issue. I should be able to put out a PR some time soon, would be great to get your eyes on it 😃 Will be sure to tag you.

[lena-kashtelyan]

An “updates” section with a date of the update on the main page would probably be very beneficial to track the changes to the website.

That’s a great idea and is actually on our roadmap for the second half of this year, so stay tuned!