fine-tune model with language model head

Hi

I would like to fine-tune esm with a language model head. I tried

import torch
model = torch.hub.load("facebookresearch/esm", "modelWithLMHead", "esm1_t34_670M_UR50S")  

but I got RuntimeError: Cannot find callable modelWithLMHead in hubconf

Is there a simple way to do this? Thanks

[joshim5]

Hi @brucejwittmann, to quickly answer your questions:

1. No, not for pretraining
2. No You asked about <pad> tokens in the batch_converter, so I brought up ignore_index as a warning just in case you implement it in a way where masks could be introduced on <pad> tokens. However, the design plan you just described sounds great and ignore_index won’t be needed in that case.

[brucejwittmann]

Hi @joshim5 , thanks for your quick and helpful response! Just to clarify on the use of ignore_index: My understanding from your paper is that loss was calculated for predictions made for masked tokens only. Does this mean that <pad> tokens were sometimes the ones that were masked? I was planning to design my masking function such that it never masks a padding token (in other words, it knows the length of each given protein and just masks amino acid tokens). If I were to do that, my understanding is that ignore_index wouldn’t be needed as <pad> could never be a target. I suppose I have a few follow-up questions, then:

1. Was loss calculated against more than the masked tokens in your original work?
2. Were <pad> tokens masked in the original work? If so, is this because there is a downside to restricting padding to amino-acid tokens only?

Thanks again!