Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

(Bug) Using 'conv_like' attention causes loss to nosedive too quickly

See original GitHub issue

edit: here’s all (edit) four runs. https://wandb.ai/afiaka87/dalle_coco_train/reports/conv_like-sanity-check--Vmlldzo1MzY3MTg

‘conv_like’ seems to still have issues at the moment.

Original post:

@lucidrains I don’t think that was the fix for conv_like. I’m still seeing the behavior from before when I turn it on.

https://wandb.ai/afiaka87/dalle_coco_train/reports/dalle-pytorch-attention-issues-using-full-axial_row-axial_col-conv_like---Vmlldzo1MzY2MjY

That’s a report of my current run. I’m gonna do a re-run with just full_conv again just to make sure it’s not due to the combination somehow.

Edit: okay, just doing a simple 16 depth, 8 heads, 8 batch size run with just ‘conv_like’ attention. Here’s that run: https://wandb.ai/afiaka87/dalle_coco_train/runs/36s8nsn9