Finue-tuning T5 model

Hi, I want to fine-tune T5 for a seq2seq task and I’m using the T5ForConditionalGeneration as it seems to have an LM decoder on top. As there’s no code example for this, I have lots of questions:

1. Am I doing the right thing?
2. I’m using the Adam optimizer. Is it ok?
3. I’m a bit confused about the forward inputs in the training phase. I read this explanation over and over again and I don’t understand whether I should just use input_ids and lm_labels for the training or not. Also somewhere in this issue someone’s mentioned that:

T5 input sequence should be formatted with [CLS] and [SEP] tokens

So which one is right? I’m super confused.

[patrickvonplaten]

Hi @amitness,

For T5 summarization you will have to append the prefix "summarize: " to every input data. But you are more or less right. All you have to do is:

1. Prepare input data

x = tokenizer.encode_plus("summarize: " + sentence, 
                          max_length=500, 
                          pad_to_max_length=True, 
                          return_tensors='pt')

1. Prepare labels

lm_labels = tokenizer.encode_plus(summary,  
                            return_tensors='pt', 
                            max_length=50, 
                            pad_to_max_length=True)

1. For tokens that are padded (which is only relevant if you train with batch_size > 1) you need to make sure that no loss is calculated on those tokens, so

lm_labels[lm_labels == tokenizer.pad_token_id] = -100

There is no need to shift the tokens as you show at the end of your comment because T5 does that automatically - see https://github.com/huggingface/transformers/blob/6af3306a1da0322f58861b1fbb62ce5223d97b8a/src/transformers/modeling_t5.py#L1063.

This is also explained in https://huggingface.co/transformers/model_doc/t5.html#training .

[enzoampil]

@amitness

E.g. in your summarization case, it would look something like:

from transformers import T5Tokenizer, T5Model

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5Model.from_pretrained('t5-small')
input_ids = tokenizer.encode("summarize: Hello, my dog is cute", return_tensors="pt")
decoder_input_ids = tokenizer.encode("<pad>", return_tensors="pt") 
outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
outputs[0]

Do note that T5ForConditionalGeneration already prepends the padding by default. Above is only necessary if you’re doing a forward pass straight from T5Model.

Regarding your question about making your own prefix, yes, you should be able to train on your own prefix. This is the whole point of T5’s text-to-text approach. You should be able to specify any problem through this kind of approach (e.g. Appendix D in the T5 paper).