Breaking change in metrics behaviour

🐛 Bug description

Following https://github.com/pytorch/ignite/pull/968 (and associated issue ) metric’s output is flatten if it returns a mapping/dict. In details, with ignite v0.4.2, for a custom metric’s we have the following:

class PerformanceIndicators(Metric):

    def compute(self):
        # ...
        return {
            'a': 12,
            'b': 23
        }

PerformanceIndicators().attach(evaluator, name="indicators")

assert "a" in evaluator.state.metrics
assert "b" in evaluator.state.metrics

# This is a breaking change introduced by the PR
assert not "indicators" in evaluator.state.metrics

print(evaluator.state.metrics)
> {'a': 12, 'b': 23}

The questions we would like to address in this issue:

• Should we fix the breaking change by readding the dict into evaluator.state.metrics such that we’ll have:

print(evaluator.state.metrics)
> {'a': 12, 'b': 23, 'indicators': {'a': 12, 'b': 23}}

• name parameter (e.g. “indicators”) is never used. Should we append it to the flatten names (e.g. indicators/a, indicators/b) or we can accept that it is never used ?

Thanks @lidq92 for reporting this.

Environment

• PyTorch Version (e.g., 1.4): 1.6.0
• Ignite Version (e.g., 0.3.0): 0.4.2
• OS (e.g., Linux):
• How you installed Ignite (conda, pip, source):
• Python version:
• Any other relevant information:

cc @sdesrozis @liebkne

[vfdev-5]

@Yevgnen thanks for pointing out about that ! To avoid duplicated plots, for now it is possible to specify which metrics to plot instead of using “all” :

wandb_logger.attach_output_handler(
    ...
    metric_names=["nll", "accuracy"],
)

• https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.wandb_logger.WandBLogger

Seeing the Wandb logger’s code, we log everything: https://github.com/pytorch/ignite/blob/9230a7319047b37ce19d956e024fa1b86030c30a/ignite/contrib/handlers/wandb_logger.py#L271-L275

however, we check the data type for others (Visdom, MLflow, Polyaxon, ClearML, …), e.g. Neptune https://github.com/pytorch/ignite/blob/9230a7319047b37ce19d956e024fa1b86030c30a/ignite/contrib/handlers/neptune_logger.py#L344-L351

Probably, we would like to make that uniform for all loggers …

[vfdev-5]

I’ll create new issue for uniform approach between loggers.