[Question] Understanding a poor training on the ADAM dataset

Hi! Awesome work 😃

Recently we have trained the nnDetection on the ADAM challenge, i.e., Task019FG_ADAM. However, the predictions on the test set are pretty bad - a lot of false postives and general sensitivity (approaching) 0. We are trying to understand where it went wrong, maybe you could be of help.

1. In your case, did the network generate a low resolution model for the ADAM challenge? Our network did end up generating a low resolution model, which we did not specifically use further on.

2. Do you have any suggestions on what could be different with your run?

The input data was unchanged apart from the omission of one patient due to having a T1 image, and we did not deviate from the instruction steps. We trained all five folds and performed a sweep for all. After that we ran the consolidation and prediction arguments as instructed.

Thank you for your help!

Best, Aaron

---

Environment Information

Currently using an NVIDIA GeForce RTX 2080 Ti; PyTorch 1.8.0; CUDA 11.2. nnDetection was installed from [~docker~ | source].

PyTorch Version: <module 'torch.version' from '/opt/conda/lib/python3.8/site-packages/torch/version.py'>
PyTorch CUDA: 11.2
PyTorch Backend cudnn: 8100
PyTorch CUDA Arch List: ['sm_52', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_86']
PyTorch Current Device Capability: (7, 5)
PyTorch CUDA available: True
System NVCC: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

System Arch List: 7.5
System OMP_NUM_THREADS: 1
System CUDA_HOME is None: False
Python Version: 3.8.5 (default, Sep  4 2020, 07:30:14) 
[GCC 7.3.0]

[mibaumgartner]

Since we ran your first setup in our experiments (both our MICCAI and nnDetection submission) I don’t think the setup will change the overall results drastically (nnDetection metrics will be worse since treated aneurysms are pretty easy to detect but I wouldn’t expect huge differences in the performance of untreated aneurysms).

From your original run (random split, treated and untreated aneurysms as foreground):

1. Can you check the metrics provided by nnDetection in the following file: [model dir]/consolidated/val_results/results_boxes.json specifically the values for AP_IoU_0.10_MaxDet_100 or FROC_score_IoU_0.10 are of interest. If the file/folder does not exist in consolidated, please run nndet_eval with the fold set to -1 (make sure the task ids are correct though and there is no mixup with the new run)
2. The file [model dir]/consolidated/val_results/pred_hist_IoU@0_1.png contains a probability histogram. The title of the figure shows how many “pos” instances there are. This value corresponds to the total number of instances in the data set and thus is great to check for debugging. (the plot of my run shows 156)
3. If you evaluate the predictions with the official ADAM script, could you eventually post your script to convert the predictions from nnDetection into the final ADAM format?

[AaronSaam]

Quick update: Following the ADAM evaluation method, the best model checkpoint predictions give a sensitivity of 0.63 and a FPcount of 0.32. I uploaded the loss curves for the five folds over here, could you check with your log files whether they look somewhat similar?

Code to plot the curves:

import matplotlib.pyplot as plt

def get_loss(data : list, find_query : str):
    output = []
    query = [x for x in data if find_query in x]
    
    for line in query:
        idx = line.find(find_query) + len(find_query)
        output.append(float(line[idx:].rstrip()))
    
    return output


task = 'Task019FG_ADAM'
network = 'nnDetection'
fold = 2

for fold in range(0,5):
    # path_data = r'D:\nnDetection\Task019FG_ADAM\log\fold4\train.log'
    
    # Path to .log file -- do make use of the variables above!
    path_data = rf'D:\{network}\{task}\log\fold{fold}\train.log'
    
    # Path to save loss curve image
    path_output = rf'D:\{network}\{task}\log\fold{fold}.png'
    
    path_data = Path(path_data)
    
    with open(path_data, 'r') as file:
        data = file.readlines()
    
    train_loss = get_loss(data, 'Train loss reached: ')
    val_loss   = get_loss(data, 'Val loss reached: ')
    #dice       = get_loss(data, 'Proxy FG Dice: ')
    
    fig, ax = plt.subplots()
    plt.title(f'{network} - {task} - fold {fold}')
    ax.plot(range(len(val_loss)), val_loss)
    #plt.plot(range(len(val_loss)), dice)
    ax.plot(range(1, len(val_loss)), train_loss)
    ax.legend(['Validation', 'Training'])
    ax.set_xlabel('Epoch')
    ax.set_ylim([-0.9, 3.4])
    #ax.set_xlim([0, 60])
    
    plt.savefig(path_output, dpi=150)