MAPE Memory Leak

The MAPE = torchmetrics.MeanAbsolutePercentageError().cuda() has a memory leak.

This is my first time using torchmetrics. After much debugging, I have isolated MAPE as the source of the problem. The only reason this loop is written so explicitly is to isolate exactly where the issue is.

sL1_list       = []
mae_list       = []
mse_list       = []
mape_list      = []

for e in tqdm(range(100)):
    
    torch.cuda.empty_cache()
    gc.collect()
    
    SL1  = torch.nn.SmoothL1Loss()
    MAE  = torch.nn.L1Loss()
    MSE  = torch.nn.MSELoss()
    MAPE = torchmetrics.MeanAbsolutePercentageError().cuda()
    
    loss           = None
    logits         = None
    spec           = None
    lld            = None
    temp_sL1_list  = []
    temp_mae_list  = []
    temp_mse_list  = []
    temp_mape_list = []
    
    #### TRAIN
    model.train()
    
    for i, (spec, lld, _) in enumerate(tqdm(train_dataloader)):
        
        spec, lld = spec.cuda(), lld.cuda()

        optimizer.zero_grad()
        logits = model(spec)[0][:, :396]
        loss = SL1(logits, lld)
        loss.backward()
        optimizer.step()
        
        break
    
    #### VALID
    model.eval()
        
    for i, (spec, lld, _) in enumerate(tqdm(train_dataloader)):
        
        spec, lld = spec.cuda(), lld.cuda()

        logits = model(spec)[0][:, :396]
        
        temp_sL1_list.append(SL1(logits, lld).item())
        temp_mae_list.append(MAE(logits, lld).item())
        temp_mse_list.append(MSE(logits, lld).item())
#         temp_mape_list.append(MAPE(logits, lld).item())
        
        break

Commenting out the one line for MAPE makes the memory leak go away.

If you look at gc.get_objects(), there is a ton of garbage made by MAPE that I don’t know how to get rid of.

Moving gc.collect() after the MAPE definition seems to fix the issue. We should not be required to re-instantiate the loss every epoch and garbage collect immediately after it is defined.

[jhkonan]

Thank you for this explanation. Everything is much clearer now — you do great work!

[justusschock]

Eval-Mode of the model does not prevent it from gradient calculation, it only changes the way certain layers work. You probably still want to wrap a with torch.no_grad around it.

The reset is required for every metric, as you can see in our quickstart tutorial here