Question on Prefix tuning code

Hi, I am looking at prefix tuning code…I have few queries on the implementation.

1. what exactly are the variables in these lines? I understand that prefix tuning provides input to every layer of the encoder-decoder model…But my understanding is that there should be a single wte and control_trans; not sure what the variables in the highlighted lines do.
2. I dont understand why the *2 in this line of code?
3. What does the control_trans variable mean in the code? what is its function?
4. Also, I see another variable mid_dim. What is it conceptually?

Thank you

[ChenWu98]

Yes, wte is for the cross-attention. We did not use “word initialization” for the prefix because they showed the benefits of such initialization under a low-data setting with only 100 samples. Actually, I am not sure, in Lisa’s paper, whether this “word initialization” was used jointly with the re-parameterization trick or only used for the embedding-only ablation. If you have any ideas, please let me know!

Yes, we assume num_encoder_layers == num_decoder_layers.

The permutation operation is used to make the tensor shape compatible with that of the key-value pairs.

[ChenWu98]

Hi, Thanks for your interest!

Our prefix-tuning code is a cleaned-up version of Lisa’s original implementation for BART. Answers to your questions are provided below, but we also recommend you look for more details in Lisa’s implementation and paper.

1. Separate prefix is learned for each attention: encoder attention, decoder attention, and cross attention.
2. *2 means the dimension of the attention key plus the dimension of the attention value.
3. control_trans is part of the re-parameterization trick introduced in Lisa’s paper.
4. mid_dim is also part of the re-parameterization trick introduced in Lisa’s paper.