Loss metric to use required_output_keys

🚀 Feature

Currently, if we have custom metrics that require data other then y_pred and y, we suggest to do the following:

metrics = {
    "Accuracy": Accuracy(),
    "Loss": Loss(criterion, output_transform=lambda out_dict: (out_dict["y_pred"], out_dict["y"])),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

where CustomMetric is defined as

class CustomMetric(Metric):

    required_output_keys = ("y_pred", "y", "x")

The idea is to extend this for Loss metric to support required_output_keys. The main issue with Loss now is with (prediction, target, kwargs) optional input, where kwargs is a dict for extra args for criterion function.

[01-vyom]

Yes, it is clear. I will change the argument and add a demo code as well as a statement that we accept a dictionary argument to the docs. I will make a PR soon. Thank you for explaining the issue.

[vfdev-5]

@01-vyom thanks for the time on studying this issue. I agree it is not clearly stated what we would like to do here. Sorry about that.

The only thing to do here is to update current implementation of Loss by defining required_output_keys = ("y_pred", "y", "criterion_kwargs") instead of None and updating the docs saying that we can now interpret output’s keys if it is a dictionary like here : https://pytorch.org/ignite/metrics.html#ignite.metrics.Metric.required_output_keys

The main idea is too simplify the code:

• BEFORE

metrics = {
    "Accuracy": Accuracy(),
    "Loss": Loss(criterion, output_transform=lambda out_dict: (out_dict["y_pred"], out_dict["y"])),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

• AFTER

metrics = {
    "Accuracy": Accuracy(),
    "Loss": Loss(criterion),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

And if we are in a use-case where user’s criterion requires some kwargs: criterion(y_pred, y, **kwargs) then our code should work almost like you suggested above:

metrics = {
    "Accuracy": Accuracy(),
    "Loss": Loss(criterion),
    "CustomMetric": CustomMetric()
}

# global criterion kwargs
criterion_kwargs = {...}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred, "criterion_kwargs": criterion_kwargs}
)

Please, let me know if it stille unclear ?