Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Loss:nan
See original GitHub issueI run train.py with all default (fcn8s and voc dataset). Here comes that Loss: nan all the time. And the val eval output is always
Overall Acc: 0.006510795767016046
Mean Acc : 0.047619047619047616
FreqW Acc : 4.239046151979406e-05
Mean IoU : 0.00031003789366743073
So where’s the problem… @_@
Issue Analytics
- State:
- Created 5 years ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
Deep-Learning Nan loss reasons - python - Stack Overflow
You may have an issue with the input data. Try calling assert not np.any(np.isnan(x)) on the input data to make sure you are...
Read more >Getting NaN for loss - General Discussion - TensorFlow Forum
Hi! The problem is not in the concatenation layer but in how you normalize the input data and how you pass it to...
Read more >Common Causes of NANs During Training
Common Causes of NANs During Training · Gradient blow up · Bad learning rate policy and params · Faulty Loss function · Faulty...
Read more >Debugging a Machine Learning model written in TensorFlow ...
The funny thing is that just before it “diverges” with loss = NaN, the model hasn't been diverging at all, the loss has...
Read more >Keras Sequential model returns loss 'nan'
I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@adam9500370 hello， I trained UNet using PASCAL VOC 2012. I found the loss keep unchanged after serval(<5) epoch. And after that the model output of train and valid always be 0. I cound not found why? Change the learning rate seem not work…
@adam9500370 hi，I have tried to train with python train.py --arch FCN8s --dataset camvid，the training process is wright. But the test result is very bad. I do not why