Problem with multirun in hydra

Hello 👋!

I am facing some issues when launching my experimentations with hydra multirun hydra -m/--multirun.

First, I noticed the configuration (omegaconf) was not visible in the dashboard when multirun is used. To reproduce:

# main.py
import hydra
from clearml import Task
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_name="config")
def my_app(cfg : DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))
    task = Task.init(project_name='hydra-clearml', task_name='hydra-clearml', reuse_last_task_id=False)

if __name__ == "__main__":
    my_app()

pip install hydra-core clearml
python main.py +test.test=1  # omegaconf logged
python main.py -m +test.test=1,2,3,4  # omegacong not captured

I wanted to force the configuration, so I added task.connect_configuration(OmegaConf.to_container(cfg, resolve=True)). I later realized it would not completely resolve the issue as all runs are stored under the same task and therefore only keeping the latest config. This is a plot taken from the task which has 4 runs. It is not a comparison between tasks, it’s the same task with all runs!

Solution

I think the best solution (and expected behaviour) would be for clearml to create one task per hydra run. This solution has the benefit of enabling performance evaluation under several hyper parameters.

[antoinebrl]

It is looking good! Runs are broken down into several tasks and the omegaconf generated by hydra is correctly captured. Well done!

[bmartinn]

I like the idea of the warning message which recommends creating and closing the task inside the app when multirun is activated.

No need to explicitly call close, only the Task.init() is important (the auto-close should be handled automatically if a multi-run is used)

Regrading the omegaconf issue, good news, I was able to reproduce the issue with multi-run. A fix is on its way 😉 I’ll update here soon