Document how each sampler treat pruned trials and failed trials

Motivated by useful discussions at https://github.com/optuna/optuna/issues/1552.

Motivation

How each sampler in Optuna treat pruned trials and failed trials is different depending on each sampler. It is kind for users to document how pruned or failed trials are processed when suggesting. I think it is a good idea to give the ..note:: section for each sampler’s docstrings.

For each sampler, the behavior is as follows.

optuna.samplers.RandomSampler & optuna.samplers.GridSampler & optuna.integration.PyCmaSampler

These samplers treat pruned trials and failed trials samely. They do not consider any pruned or failed trials. They simply ignore those trials.

optuna.samplers.TPESampler

This sampler treats pruned trials and failed trials differently. This sampler simply ignores failed trials. On the other hand, this sampler considers pruned trials to suggest the next parameters in each iteration. Concretely, This sampler makes a ranking of completed and pruned trials based on the pairs of the completed or pruned step and the evaluation value when completed or pruned. Then, this sampler suggests the next parameters according to the priority of the trial based on the ranking.

optuna.samplers.CmaEsSampler & optuna.integration.SkoptSampler

These samplers treat pruned trials and failed trials differently. This sampler simply ignores failed trials. On the other hand, this sampler considers pruned trials only when the consider_pruned_trials flag is True. When consider_pruned_trials = True, these samplers consider that pruned trials’ evaluation values are the evaluation values when pruned.

[HideakiImamura]

Oh! Thanks, @c-bata. Fixed.

[fsmosca]

I also observed that in SkoptSampler with considered_pruned_trials=True, the param of pruned trial is repeated. So there must be something with this considered_pruned_trials.