NSGAIIStoppingByTimeRunner Experiment in JMetal?

I am performing experiments with JMetal and I have to say that metal’s experiments are very useful, but my interest is to use StoppingByTime criteria instead of maxEvaluations.

We imagine that we want to perform the NSGAIIStudy-like experiments, but instead of using NSGAII we want to use NSGAIIStoppingByTime and that each instance lasts 1 minute(int thresholdComputingTimeInMilliseconds = 60000;)

• NSGAIIStudy -> https://github.com/jMetal/jMetal/blob/master/jmetal-exec/src/main/java/org/uma/jmetal/experiment/NSGAIIStudy.java
• NSGAIIStoppingByTime -> https://github.com/jMetal/jMetal/blob/master/jmetal-algorithm/src/main/java/org/uma/jmetal/algorithm/multiobjective/nsgaii/NSGAIIStoppingByTime.java

It is easy to change the calls of the NSGAII algorithm to NSGAIIStoppingByTime in the experiments template, But…

TROUBLE

After the execution, the duration of the entire experiment of all instances and INDEPENDENT_RUNS only lasts (1 minute = 60000 ms) as long as I want each of the INDEPENDENT_RUNS to last.

It seems that the system does not recognize that each instance has its independent thresholdComputingTimeInMilliseconds variable.

In other words, if I want to experiment with 3 algorithms, 25 INDEPENDENT_RUNS and each run lasts 1 minute, the experiment should last 3X25X1 = 75 minutes (or a little less for parallelism, I guess), however, the experiment only lasts 1 minute in total (the thresholdComputingTimeInMilliseconds value)

CODE EXAMPLE

Thanks in advance. mmahrach

[matthieu-vergne]

This issue is a generic one, not just for our case. Whatever software you run on whatever hardware, time limits never guarantee strict reproducibility because of random variables. If having a comparable number of evaluations is important for you, then your best choice is to run for a given amount of evaluations. Nobody can guarantee both a similar time and a similar number of evaluations.

I don’t minimize this issue, which clearly shows problems in our code. But if your concern is the comparability, which is a good thing, then time limits are recognised as bad practices.

[matthieu-vergne]

Since you know it should run in total around 75 minutes, you may set the limit to 75 minutes. Of course it is not a proper solution, but it seems to me to be the closest and simplest workaround.

The issue with time computation is that you cannot guarantee the determinism. We do have some randomization processes, but we can use the same seeds to ensure we obtain the same results. With time computation, we cannot, since the computation effort is distributed among threads and other processes on the machine based on the decision of the processor, which may vary between runs. In particular, time limits are always approximative since the processors itself cannot guarantee to stop exactly at the requested nanosecond. Moreover, the machine may be overloaded at some point, but the impacted runs will nevertheless stop after the given time.

Although it does not completely discard the use of time limits, it does support the fact that we cannot have reliable experiments with them. In other words, even if we do solve the issue you highlighted here, at best I expect it to be just a bit more precise than the workaround I suggest at the beginning of this post. So the effort to fix it might be unworthy.