Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Absolute positional embedding present with ALiBi relative positional encoding.

See original GitHub issue

Hi. To the best of my understanding, this line of code should be like this:

self.has_pos_emb = position_infused_attn or rel_pos_bias or rotary_pos_emb or alibi_pos_bias

https://github.com/lucidrains/x-transformers/blob/a9de3a837ae69c917d8e26e71b967f750be6a1d2/x_transformers/x_transformers.py#L675

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:7 (5 by maintainers)

Top GitHub Comments

1reaction

lucidrainscommented, Nov 29, 2021

@Atze00 oh yea, that works 👍

1reaction

Atze00commented, Nov 29, 2021

Isn’t the flag already present? use_pos_emb should work fine. This definition for the transformer-XL should work also for the ALiBi positional encoding:

model_xl = TransformerWrapper(
    num_tokens = 20000,
    max_seq_len = 512,
    max_mem_len = 2048,
    use_pos_emb = False,
    attn_layers = Decoder(
        dim = 512,
        depth = 6,
        heads = 8,
        rel_pos_bias = True,
        pre_norm = False
    )
)

Top Results From Across the Web

Attention with Linear Biases Enables Input Length Extrapolation.

When using ALiBi, we do not add positional embeddings at the bottom of the network. 3 ATTENTION WITH LINEAR BIASES (ALIBI). In the...

Relative Positional Encoding - Jake Tae

In this post, we will take a look at relative positional encoding, as introduced in Shaw et al (2018) and refined by Huang...

[PDF] CAPE: Encoding Relative Positions with Continuous ...

This paper proposes an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute and relative ...

Relative Position Encodings Explained - Papers With Code

In other words, instead of simply combining semantic embeddings with absolute positional ones, relative positional information is added to keys and values on ......

Master Positional Encoding: Part II | by Jonathan Kernes

Whereas the first article discussed the meaning of the fixed sinusoidal absolute positional encodings, this article will focus on relative ...