Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Adaptive Pixel Intensity Loss generated NaN values while training

See original GitHub issue

Was training on custom human dataset. Batch Size = 8 No of training images = 3800

No of steps trained before showing error = 75

After 75th step It generated an error:

RuntimeError: Function 'UpsampleBilinear2DBackward1' returned nan values in its 0th output.

The model trained successfully when using BCE loss.

We even checked for NaN values using torch.autograd.set_detect_anamoly(True) But it returned False stating that no NaN values were found

Issue Analytics

State:
Created 2 years ago
Comments:10 (4 by maintainers)

Top GitHub Comments

1reaction

Karel911commented, Jan 28, 2022

@Karel911 can you help me with removing the edge generation parts? because i am facing a similar issue.

I also curious about which parts make this issue. I released the version of TRACER without edge generation. Replace the released scripts with the existing ones. I briefly tested it so if there is any problem, please let me know.

Thanks.

0reactions

hackkhaicommented, Jan 28, 2022

Thanks, Let me check this out