Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Extremely hard incremental scenario

See original GitHub issue

Thank you for your great work, PLOP!

As your code, we can reproduce the performance almost the same as your paper.

Additionally, we also conduct experiments on extremely hard incremental scenarios, such as 5-1 (16 steps) and 2-1 (19 steps).

for this, we add below lines on task.py

"5-1": {
        0 : [0, 1, 2, 3, 4, 5],
        1 : [6, ],
        2 : [7, ],
        3 : [8, ],
        4 : [9, ],
        5 : [10, ],
        6 : [11, ],
        7 : [12, ],
        8 : [13, ],
        9 : [14, ],
        10: [15, ],
        11: [16, ],
        12: [17, ],
        13: [18, ],
        14: [19, ],
        15: [20, ],
    },   
    "2-1":{
        0 : [0, 1, 2],
        1 : [3, ],
        2 : [4, ],
        3 : [5, ],
        4 : [6, ],
        5 : [7, ],
        6 : [8, ],
        7 : [9, ],
        8 : [10, ],
        9 : [11, ],
        10: [12, ],
        11: [13, ],
        12: [14, ],
        13: [15, ],
        14: [16, ],
        15: [17, ],
        16: [18, ],
        17: [19, ],
        18: [20, ],
    },

However, during the training, the loss is divergence to nan.

I already noticed that someone suffers from the loss divergence issue https://github.com/arthurdouillard/CVPR2021_PLOP/issues/8 on 15-5 task, however, I can reproduce the performance on 15-5 task in my environmental settings. Also, MiB on these extremely hard scenarios was well trained without the loss divergence, however, PLOP showed the issue.

Therefore, I wonder you also have the same issue in the extremely hard scenarios, 5-1 and 2-1. And, please tell me how can I solve that issue (e.g., which hyperparameter should be changed).

Thanks.