Is it possible to use AutoregressiveWrapper in combination with ContinuousTransformerWrapper

Can I combine the AutoregressiveWrapper with the ContinuousTransformerWrapper? ignore_index and pad_value are scalars and should be tensors here I believe.

Should I create a custom ContinuousAutoregressiveWrapper for this?

I’m trying to use it like this:

model = AutoregressiveWrapper(ContinuousTransformerWrapper(
    max_seq_len=self.max_sequence_length,
    dim_in=self.vector_dimension,
    dim_out=self.vector_dimension,
    emb_dim=self.embedding_dimension,
    use_pos_emb=True,
    attn_layers=Decoder(
        dim=self.embedding_dimension,
        depth=self.depth,
        heads=self.heads,
        attn_dropout=self.dropout,
        ff_dropout=self.dropout,
        rotary_pos_emb=True
    )
),
    pad_value=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]           # For example
).to(utils.get_device())

[wingedsheep]

It works! I trained a model on the midi of “Californication” converted to encodings by the auto-encoder, and it actually learns to generate the song 😄 I’m curious how it will do when trained on more data.

[lucidrains]

very cool! the faceformer paper has some interesting twists on the ALiBi encoding too to make their deep net work. maybe you can draw some inspiration from their architecture if you get stuck