Queries about the Notation and Model training of T5 and ELECTRA sentiment classification.

I have a few questions about the model notation. And also short info about T5 and ELECTRA. I would like to make separate issues but things are not too complex. I mainly working on CV, so sorry if I being so silly.

1 Cased or Uncased

What is mean by cased and uncased?

bert-base-uncased
bert-base-cased

2 Suffix

I was trying to run the XLM model but in the pre-train model, I’ve found the following weights, I understood about XML-MLM but couldn’t get the rest of the part, ex: enfr-1024, enro-1024 etc.

xlm-mlm-enfr-1024
xlm-mlm-enro-1024
xlm-mlm-tlm-xnli15-1024

3 Sentiment Analysis using T5 and ELECTRA

Is it possible to use these two models for sentiment classification, simply just a binary classification? How can we implement these two transformers? I have a high-level overview of T5, it transforms both (input/target) as a text. I found it useful though but bit trouble to implement. Using transformers, is it possible to go within a convenient way?

[craffel]

Hi, it is easy to use the pre-trained T5 models for sentiment ID. You could do something like

MODEL_NAME = "t5-base"
model = transformers.T5ForConditionalGeneration.from_pretrained(MODEL_NAME)
tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)
input_text = "sst2 sentence: This movie was great! I loved the acting."
inputs = tokenizer.encode_plus(input_text, return_token_type_ids=False, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs)[0]))
input_text = "sst2 sentence: The acting was so bad in this movie I left immediately."
inputs = tokenizer.encode_plus(input_text, return_token_type_ids=False, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs)[0]))

The "sst2 sentence:" prefix is what we used for the SST-2 task. It is a sentiment ID task. The model needs to see this prefix to know what task you want it to undertake.

[craffel]

I sure I’m missing something crucial part that is not considering text-to-text manner. If I convert 1 and 0 of labels as Positive and Negative…I mean shouldn’t the target need to be numeric!

No, the target should always be text for T5. You should map your 0/1 labels to the words “negative” and “positive” and fine-tune T5 to predict those words, and then map them back to 0/1 after the model outputs the text if needed. This is the point of the text-to-text framework - all tasks take text as input and produce text as output. So, for example, your “build model” code should not include a dense layer with a sigmoid output, etc. There is no modification to the model structure necessary whatsoever.

And about the prefix, sst2 sentence: so, this is, in other words, is a string indicator to inform the model about the goal or task. So, do I have to add this string at the beginning of every text sentence or (samples)?

Yes, that is the intention.