unexptected behavior of AdamP on CPU?

Hi, I’m wondering whether AdamP optimizer may have some unexpected behavior on cpu.

I trained my models of the same parameters with Adamp on cuda and cpu. Adamp gives excellent outcomes when it is run on the cuda but, interestingly, not on the cpu. More precisely, it was not able to minimize train/val loss.

Here’s a code snippet that I used to train my model. https://github.com/Junyoungpark/PGNN/blob/main/train_pgnn.py

[bhheo]

Thank you for your valuable comments 👍 I will fix it when I find a clue for the problem.

[bhheo]

Oh, then, it is clear that AdamP causes the problem.

I’m sorry but, it is really hard to figure out the reason for cpu issue. I have read the code multiple times. Everything looks just fine.

I’m suspicious of the built-in cosine functionF.cosine_similarity. You can try an old version of AdamP(https://github.com/clovaai/AdamP/blob/v0.2.0/adamp/adamp.py) to verify the built-in cosine function. If the old version has the same problem, I have no idea what is causing the problem.