HPO not terminate as expect `max_experiment_duration`

Describe the issue:

How to understand experiment.config.max_experiment_duration = "1m" option? The HPO process keeps running although reaching DDL.

I just use the example configuration:

SEARCH_SPACE = {
    "lr": {"_type": "qloguniform", "_value": [1e-4, 1, 1e-4]},
    "momentum": {"_type": "quniform", "_value": [0.1, 0.99, 0.01]},
    "weight_decay": {"_type": "qloguniform", "_value": [1e-5, 1e-2, 1e-5]},
    "batch_size": {"_type": "choice", "_value": [32, 64, 128, 256, 512]},
}

experiment = Experiment("local")
experiment.config.experiment_name = "Cifar Test"
experiment.config.trial_concurrency = 4
experiment.config.max_trial_number = 10
experiment.config.trial_gpu_number = 1
experiment.config.search_space = SEARCH_SPACE
experiment.config.max_experiment_duration = "1m"
experiment.config.trial_command = "python cifar.py"
experiment.config.tuner.name = "TPE"
experiment.config.tuner.class_args["optimize_mode"] = "maximize"
experiment.config.training_service.use_active_gpu = True
experiment.config.training_service.gpu_indices = [0, 1, 2, 3]

experiment.run(8080)

Environment:

• NNI version: 2.9
• Training service: local
• Client OS: Unbuntu 20.04
• Python version: 3.9
• PyTorch/TensorFlow version: Pytorch 1.12
• Is conda/virtualenv/venv used?: conda
• Is running in Docker?: No

[Lijiaoa]

no_more_trial is not the same as the experiment status Running. It means tuner could not dispatch trial because this experiment had reached max duration 1min. So I suggest you set max_duration to 1h and re-run the experiment.

[Lijiaoa]

https://nni.readthedocs.io/en/stable/reference/nnictl.html#nnictl-log I think the doc is useful for you.