Training Classification Model + tokenizer from Scratch

Describe the bug I’ve been looking through the repository and observed the ClassificationModel requires a pre-trained model as part of the args. Is there any chance we can take a model from scratch, train a tokenizer for our dataset and train the model on classification? Any help is really appreciated, I’ve really enjoyed using the simple transformers library for my research!

[ThilinaRajapakse]

I can answer this empirically - option 2 is superior. That is why transfer learning is so powerful.

The philosophical question is whether it’s worth the effort to fine-tune a pre-trained language model on the domain-specific text (i.e. fine-tune the language model itself) before training the model on the classification itself. In this case, I would suggest skipping the language model fine-tuning at first, and then coming back to it if the final results are not satisfactory.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.