Interpretation callback - best and worst indices of a loss for loader

🚀 Feature Request

Hi, I think we already discussed this in Slack, but the basic idea is the following: there is a callback called e.g LossInterpretationCallback which shows the best/worst examples for a loss in the Tensorboard.

Motivation

Motivation: quick error analysis, to gain a quick intuition on when is the model doing great and when it is doing badly. Since it is just a callback, users can always just not include it, there is no risk for other features.

Proposal

My proposal is to create a class called LossInterpretationCallback located either under contrib or under dl.callbacks, that for every loader that uses a SequentialSampler (otherwise we cannot know the original index in the dataset) computes a given criterion reduced per sample, and then sorts it, and plot top k / bottom k examples to the Tensorboard, and also serializes the interpretations for example in a loss_interpretation directory, so that they can be further inspected later.

Alternatives

Several similar, but slightly different callbacks exist:

• ShowPolarBatchesCallback in pytorch-toolbelt - shows only the best one and worst one batches, not samples. Slightly different, still very useful
• InterpretationCallback in dnn.cool - the same thing, but it forces you to use multi-task learning classes, is awkward if you have just one output.

Additional context

Here’s for example how would the minimal classification example change:

#%%

import torch
from torch.utils.data import DataLoader, TensorDataset
from catalyst import dl

#%%
# sample data
num_samples, num_features, num_classes = int(1e4), int(1e1), 4
X = torch.rand(num_samples, num_features)
y = (torch.rand(num_samples, ) * num_classes).to(torch.int64)

# pytorch loaders
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=32, num_workers=1)
loaders = {"train": loader, "valid": loader}

# model, criterion, optimizer, scheduler
model = torch.nn.Linear(num_features, num_classes)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [2])

#%%
callbacks = [dl.AccuracyCallback(num_classes=num_classes),
             dl.LossInterpretationCallback(torch.nn.CrossEntropyLoss(reduction='none'))]
#%%
# model training
runner = dl.SupervisedRunner()
runner.train(
    model=model,
    criterion=criterion,
    optimizer=optimizer,
    scheduler=scheduler,
    loaders=loaders,
    logdir="./logdir",
    num_epochs=3,
    callbacks=callbacks
)

[Scitator]

In this case looks like we really could implement such useful callback - it would be great, btw, @bagxi @Ditwoo could you please share your thoughts on this feature and check our proposals?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.