Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PerceiverResampler missing some LayerNorms?

See original GitHub issue

Hey, it feels like PerceiverResampler is missing some LayerNorms? it seems to me we should layer-norm x before sending to attentions loop, and may be add layer-norm to ff(latents) + latents?

Issue Analytics

State:
Created a year ago
Reactions:3
Comments:7 (5 by maintainers)

Top GitHub Comments

1reaction

lucidrainscommented, Apr 28, 2022

Indeed I’ll add it soon

Lol, just finished building it

0reactions

inspiritcommented, Apr 30, 2022

I am talking about x here: https://github.com/lucidrains/flamingo-pytorch/blob/932d2a0997b436d7c6c91d29c92d989d4fb80000/flamingo_pytorch/flamingo_pytorch.py#L104

you pass it later to different attention layers without it being modified so its probably make sense to simply norm it once after adding time_pos_emb

Top Results From Across the Web

How are Learned Latent Arrays for the Perceiver Resampler in ...

My understanding is that these learned latent arrays are a reduced dimensional representation of the visual feature arrays that are the outputs ...

ICLR 2022 Conference - OpenReview

... target variables given feature vectors with some of the elements missing. ... Our model augments the Perceiver with a flexible querying mechanism...

lucidrains/perceiver-pytorch: Implementation of ... - GitHub

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch - GitHub ... As an example, using PerceiverIO as a language model.

The Annotated Perceiver. A detailed PyTorch tutorial for the…

In this article I will explain DeepMind's Perceiver architecture and provide a thoroughly annotated working implementation. This is intended to be Part 1...

LayerNorm — PyTorch 1.13 documentation

LayerNorm (normalized_shape, eps=1e-05, elementwise_affine=True, device=None, ... Applies Layer Normalization over a mini-batch of inputs as described in the ...