Pad token for GPT2 and OpenAIGPT models

❓ Questions & Help

I noticed that out of all the models pad_token is not set for only OpenAIGPTModel and GPT-2Model. I get a warning: Using pad_token, but it is not set yet. and pad_token_id is None

Is there any specific reason why is that so?

If not, what is the appropriate padding token to be used for these models?

Thanks

[Choitsugun]

Because GPT2 and GPT are causal LM you don’t need to pad shorter sentences in batches. Why? The pad is to make up the length of the batch. Does this have anything to do with GPT2’s causal model?

[patrickvonplaten]

Because GPT2 and GPT are causal LM you don’t need to pad shorter sentences in batches. It is important though that the loss on these “unnecessary” tokens is not calculated. You should set all lables corresponding to “PADDED” tokens to -100. In the code snippet you can see in the map_to_encoder_decoder_inputs function how the labels are set to -100 for attention_mask = 0: https://huggingface.co/patrickvonplaten/bert2gpt2-cnn_dailymail-fp16#training-script