Interpretation callback - best and worst indices of a loss for loader
See original GitHub issue🚀 Feature Request
Hi, I think we already discussed this in Slack, but the basic idea is the following: there is a callback called e.g LossInterpretationCallback
which shows the best/worst examples for a loss in the Tensorboard.
Motivation
Motivation: quick error analysis, to gain a quick intuition on when is the model doing great and when it is doing badly. Since it is just a callback, users can always just not include it, there is no risk for other features.
Proposal
My proposal is to create a class called LossInterpretationCallback
located either under contrib
or under dl.callbacks
, that for every loader that uses a SequentialSampler
(otherwise we cannot know the original index in the dataset) computes a given criterion reduced per sample, and then sorts it, and plot top k / bottom k examples to the Tensorboard, and also serializes the interpretations for example in a loss_interpretation
directory, so that they can be further inspected later.
Alternatives
Several similar, but slightly different callbacks exist:
- ShowPolarBatchesCallback in
pytorch-toolbelt
- shows only the best one and worst one batches, not samples. Slightly different, still very useful - InterpretationCallback in
dnn.cool
- the same thing, but it forces you to use multi-task learning classes, is awkward if you have just one output.
Additional context
Here’s for example how would the minimal classification example change:
#%%
import torch
from torch.utils.data import DataLoader, TensorDataset
from catalyst import dl
#%%
# sample data
num_samples, num_features, num_classes = int(1e4), int(1e1), 4
X = torch.rand(num_samples, num_features)
y = (torch.rand(num_samples, ) * num_classes).to(torch.int64)
# pytorch loaders
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=32, num_workers=1)
loaders = {"train": loader, "valid": loader}
# model, criterion, optimizer, scheduler
model = torch.nn.Linear(num_features, num_classes)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [2])
#%%
callbacks = [dl.AccuracyCallback(num_classes=num_classes),
dl.LossInterpretationCallback(torch.nn.CrossEntropyLoss(reduction='none'))]
#%%
# model training
runner = dl.SupervisedRunner()
runner.train(
model=model,
criterion=criterion,
optimizer=optimizer,
scheduler=scheduler,
loaders=loaders,
logdir="./logdir",
num_epochs=3,
callbacks=callbacks
)
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:8 (4 by maintainers)
In this case looks like we really could implement such useful callback - it would be great, btw, @bagxi @Ditwoo could you please share your thoughts on this feature and check our proposals?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.