Config specifies max_position_embeddings as 1024

Hi!

I noticed that the PRIMERA configs specifies max_position_embeddings: 1024. Is this intentional? AFAICT the HuggingFace library treats this as the maximum position embedding size of the encoder, or max_encoder_position_embeddings, which for PRIMERA is 4096.

E.g. in their run_summarization.py script, they appear to treat max_position_embeddings as max_encoder_position_embeddings as they compare it to the max_source_length.

So I am wondering if max_position_embeddings should be set to 4096 in the PRIMERA configs, else it causes problems when trying to use with existing HF example scripts.

[Wendy-Xiao]

Hi John and Jay,

Thanks for pointing out the issue. It is not intentional to set max_position_embedding to be 1024, it’s just set by default.

I have not used run_summarization.py before, so I didn’t know that in that script they treat max_position_embedding as the maximum position embedding size for the encoder, which is specified as max_encoder_position_embeddings in our case.

I’ll update it in the config files, to make it consistent with the script.

[jayded]

I can confirm that this affects run_summarization.py and is inconsistent with the semantics of other Huggingface configs.