Possible bug in latent vector loss calculation?

I’m confused by this, and wondering if it could be a bug? It seems as though latents is of size (32,128), which means that for array in latents: iterates 32 times. However, the results from these iterations aren’t stored anywhere, so they are at best a waste of time and at worst causing a miscalculation. Perhaps the intention was to accumulate the kurtoses and skews for each array in latents, and then computing lat_loss using all the accumulated values?

for array in latents:
    mean = torch.mean(array)
    diffs = array - mean
    var = torch.mean(torch.pow(diffs, 2.0))
    std = torch.pow(var, 0.5)
    zscores = diffs / std
    skews = torch.mean(torch.pow(zscores, 3.0))
    kurtoses = torch.mean(torch.pow(zscores, 4.0)) - 3.0

lat_loss = lat_loss + torch.abs(kurtoses) / num_latents + torch.abs(skews) / num_latents

Occurs at https://github.com/lucidrains/big-sleep/blob/main/big_sleep/big_sleep.py#L211

[walmsley]

That fully addresses the above points, thanks!

BUT also perhaps most significantly, there is a deeper possible bug that I’m curious about. I was trying to understand why num_latents is set to 32 here: https://github.com/lucidrains/big-sleep/blob/31fa846e0e300515360fbeb67a0315f20505fd59/big_sleep/big_sleep.py#L89-L92

Diving deeper, it seems as though these 32 different vectors are only actually used within cond_vector here: https://github.com/lucidrains/big-sleep/blob/31fa846e0e300515360fbeb67a0315f20505fd59/big_sleep/biggan.py#L510 and here: https://github.com/lucidrains/big-sleep/blob/31fa846e0e300515360fbeb67a0315f20505fd59/big_sleep/biggan.py#L520

I debugged the loop surrounding line 520 above (using the current 512px BigGAN model), and found that the model only actually contains 15 layers; of those 15, only 14 of those layers are GenBlock layers which trigger line 520.

The result is that of the 32 latent vectors we create, only indices {0,1,2,3,4,5,6,7,8,10,11,12,13,14,15} are actually ever used. This wouldn’t be a problem, except that the remaining 17 unused latent vectors may still be influencing the loss calculation. I’m still trying to work out whether their influence on the loss calculation is significant enough to merit fixing this, because the fix would be slightly nontrivial as it varies depending on the size of BigGAN model chosen.

[walmsley]

Created new PR with proposed final fix @ https://github.com/lucidrains/big-sleep/pull/35

Overall status of 4 possible bugs mentioned in this issue:

• kurtosis/skew accumulation (fixed in release 0.5.1)
• latent loss mean(dim=1) (fixed in release 0.5.2)
• class loss topk[0] (not a bug, no fix needed)
• num_latents 32 -> 15 (fixed in release 0.5.3)