Support custom events for metrics update

Currently for ignite metrics, update function is called per iteration. In scenarios like multitasking, there can be metrics for different tasks, and each iteration might only have outputs for a subset of tasks. In this situation, only the metrics associated with those tasks having outputs can/should be updated.

To make it work, I need to overwrite the attach function like this:

class CustomRnningAverage(RunningAverage):
    def attach(self, engine, name, custom_event=None):
        if custom_event is None:
            super().attach(engine, name)
        else:
            if self.epoch_bound:
                # restart average every epoch
                engine.add_event_handler(Events.EPOCH_STARTED, self.started)
            # compute metric
            engine.add_event_handler(custom_event, self.iteration_completed)
            # apply running average
            engine.add_event_handler(custom_event, self.completed, name)

This is doable, but would be nice to have it by default in ignite. Also, this manual overwrite doesn’t work with DDP. The cause is related to _internal_attach in metrics_lambda.py.

I would like to propose

1. Have update every iteration by default
2. Support overwrite when to update by accepting custom events
3. Make it work with DDP

Multitasking is one case where this can be very helpful. But I am sure there will be other use cases.

[sdesrozis]

Yes, MetricUsage should be the way to configure your metric.

Epoch-wise accuracy :

engine = ...
acc = Accuracy()
acc.attach(engine, "acc", usage="epoch_wise")  # default

It means reset of metric at Event.EPOCH_STARTED, update at Event.ITERATION_COMPLETED and complete at Event.EPOCH_COMPLETED

Batch-wise accuracy :

engine = ...
acc = Accuracy()
acc.attach(engine, "acc", usage="batch_wise")

It means reset of metric at Event.ITERATION_STARTED, update at Event.ITERATION_COMPLETED and complete at Event.ITERATION_COMPLETED

You can define a custom usage

my_usage = YourCustomUsage()
acc = Accuracy()
acc.attach(engine, "acc", usage= my_usage)

Here, YourCustomUsage must inherit from MetricUsage and you customize when reset, update and complete of metric occur.

HTH

[snie2012]

@sdesrozis Would you mind showing an example of using the changes you made in sdesrozis:issue_874 to update on custom events? My understanding is that I can create a customized MetricUsage class to do that.