RuntimeError: index out of range at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

🐛 Bug

Information

Model I am using (Bert, XLNet …): Bert-base-uncased

Language I am using the model on (English, Chinese …): English

The problem arises when using:

• the official example scripts: (give details below)
• [ x] my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• [x ] my own task or dataset: (give details below)

To reproduce

I am trying to fine-tune BERT for sentiment analysis on the IMDB Dataset. Most of the code is based on this blog: http://mccormickml.com/2019/07/22/BERT-fine-tuning/

I create my model as follows:

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
    num_labels = 2, # The number of output labels--2 for binary classification.
    output_attentions = False, # Whether the model returns attentions weights.
    output_hidden_states = False, # Whether the model returns all hidden-states.
)

In my training loop, I do the following:

for step, batch in enumerate(train_dataloader):
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)
     
        model.zero_grad()

       loss, logits = model(b_input_ids, 
                             token_type_ids=None, 
                             attention_mask=b_input_mask, 
                             labels=b_labels)

I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-77ffde63bb0d> in <module>()
    109                              token_type_ids=None,
    110                              attention_mask=b_input_mask,
--> 111                              labels=b_labels)
    112 
    113         # Accumulate the training loss over all of the batches so that we can

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels)
   1030                             position_ids=position_ids,
   1031                             head_mask=head_mask,
-> 1032                             inputs_embeds=inputs_embeds)
   1033 
   1034         pooled_output = outputs[1]

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734 
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    185         if inputs_embeds is None:
    186             inputs_embeds = self.word_embeddings(input_ids)
--> 187         position_embeddings = self.position_embeddings(position_ids)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    189 

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1465         # remove once script supports set_grad_enabled
   1466         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1467     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1468 
   1469 

RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

Expected behavior

Environment info

• transformers version: 2.3.0
• Platform: Mac OS
• Python version: 3.6
• PyTorch version (GPU?): 1.2.0, no GPU
• Tensorflow version (GPU?): 1.2.0, no GPU
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

[KaiserWhoLearns]

I had the same issue while my per_gpu_eval_batch_size=8 and per_gpu_train_batch_size=8 .

[sameul-yuan]

return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

RuntimeError: index out of range: Tried to access index 0 out of table with 202 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

is there anyone knows what’s the problem ? , I checked the embedding matrix of shape (203, 16), it shows index 0 out of range, how it is possible???