Train RobertaModel from scratch for my dataset

I am trying to train RobertaModel from scratch. I am following this blog but instead of model = RobertaForMaskedLM(config=config), I am starting with configuration = RobertaConfig() model = RobertaModel(configuration) and then continuing with other steps. But I am getting error TypeError: forward() got an unexpected keyword argument 'labels'. The whole code piece:

configuration = RobertaConfig()
model = RobertaModel(configuration)
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./train.txt",
    block_size=128,
)
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False, mlm_probability=0.15
)
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./Model1",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)
trainer.train()

Is there some other way to do pre-training? Am i missing something here?

[patil-suraj]

You can train RobertaForMaskedLM using the mlm objective and then load it in RoberatForSequenceClassification for classification . RoberatForSequenceClassification will take care of taking last layer vector and feeding it to a classification layer.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.