low loss in fine tuning but generated answers are not correct

Hi, I am fine tuning a QA dataset using huggingface unified v2 t5 large, and the sample code is like below

# training
model_inputs = self.tokenizer(questions,
                padding=True, truncation=True, 
                max_length=self.tokenizer.model_max_length, return_tensors="pt").to(device)
with self.tokenizer.as_target_tokenizer():
    labels = self.tokenizer(answers,
                    padding=True, truncation=True, 
                    max_length=self.tokenizer.model_max_length, return_tensors="pt").to(device)
    # ignore pad token for loss
    labels["input_ids"][
                labels["input_ids"] == self.tokenizer.pad_token_id
    ] = -100
    model_inputs["labels"] = labels["input_ids"]
outputs = self.model(**model_inputs)
loss = outputs.loss


# generate
model_inputs = self.tokenizer(questions, 
                padding=True, truncation=True, 
                max_length=self.tokenizer.model_max_length, return_tensors="pt").to(device)

sampled_outputs = self.model.generate(**model_inputs, 
                num_beams=4, max_length=50, early_stopping=True)

I can get fairly low loss (0.41) after fine tuning for around 5 epochs, yet the generated answers are mostly wrong (0.23 accuracy). According to T5 doc it seems that generate can handle the prepending of pad token. Also, the generated answers indeed belong to one of the choices, it is just that they are not the correct ones. I am wondering what might be the issue. Thanks!

[cnut1648]

Oh I see. Regardless, I think the lesson I learned is that if the performance is not correlated with the loss we can give unifiedqa a longer training epochs/steps. Thank you for the help all the way @danyaljj!!

[cnut1648]

Thanks @danyaljj! After a week’s attempt I think I somehow solved this problem. In my case, it seems that fine tuning more epochs will work. Previously I was fine tuning either 5 or 10 epochs, and got 0.23 accuracy. When fine tuning for 50 epochs, I can get 0.72 accuracy. I wonder that in your paper did you also fine tune with large epoch? Thanks!!