How can I initialize RobertaForSequenceClassification empty?

I am having my own custom dataset which is completely different from the one with roberta. Now using ByteLevelBPETokenizer, I am creating vocab.json and merges.txt . I am using these two to files to initialize RobertaTokenizerFast for encoding my corpus. Now I am training a RobertaForSequenceClassification for a binary classification problem. When I am initializing RobertaForSequenceClassification with any of the pre-trained models I am getting IndexError: index out of range in self on the RAM while on GPU I am getting RuntimeError: CUDA error: an illegal memory access was encountered . I have followed other issues but of no help. My understanding since I am creating my own vocabulary, some of the tokens are not in pre-trained model. So is there a way to initialize RobertaForSequenceClassification empty or to train this classification model for my dataset?

[SergioBarretoJr]

@SergioBarretoJr I am also facing the same issue, can you please help me?

@amandalmia14 Have you ever checked if you set vocab size equal in tokenizer and RoBERTa.config file?

def train_tokenizer(self,file_path,outDir):
    # Initialize a tokenizer
    tokenizer = ByteLevelBPETokenizer()
    # Customize training
    tokenizer.train(files=file_path, **vocab_size=52_000**, min_frequency=2, special_tokens=[
        "<s>",
        "<pad>",
        "</s>",
        "<unk>",
        "<mask>",
    ])
    self.tokenizer=tokenizer
    tokenizer.save(outDir)

This was how I solver my issue. Can I help with something more?

[LysandreJik]

Hi! If you want to initialize a new RobertaForSequenceClassification model you can do so as such:

from transformers import RobertaForSequenceClassification, RobertaConfig

config = RobertaConfig(
    # put the args you need, like the vocab size
    vocab_size=100
)

model = RobertaForSequenceClassification(config)