Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Potential EMA Model Issues
See original GitHub issueSo, I’ve noticed a potential bug related to the exponential moving averaged priors (I haven’t tested the other models).
Essentially, the EMA model seemingly refuses to “learn”. I have to believe that the weights aren’t being properly updated with the online_model, for whatever reason.
As you can see, as soon as the EMA kicks in at step 1000, the loss plateaus. I’ve never worked with an EMA model before, so I guess this could be expected behavior?
This issue also presents itself when running evaluation on the EMA model (the (text<->predicted_image) similarity should be at least 0.1 even after a few hundred steps)
Issue Analytics
- State:
- Created a year ago
- Comments:5 (2 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
@nousr oh crap, yes, you are right! i thought that i wasn’t incrementing the EMA steps in accordance with the global training step, but i was
https://github.com/lucidrains/DALLE2-pytorch/commit/9cc475f6e7990b3d978128902ee0ea90614451f6 should fix the update every issue
@lucidrains hey! when you get a chance, do you think could you explain the intuition for this line?
https://github.com/lucidrains/DALLE2-pytorch/blob/1cc288af39f171b3e7f77fdb4252682af05e17e9/dalle2_pytorch/trainer.py#L191
specifically, why the
// self.update_every? I guess I don’t see the purpose of movingself.update_after“up” by a factor ofupdate_every?