RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Here is the complete error message:

Traceback (most recent call last):
  File "app/set_expantion_eval.py", line 118, in <module>
    map_n=flags.map_n)
  File "app/set_expantion_eval.py", line 62, in Eval
    expansionWithScores = BE.set_expansion_tensorized(seeds, ["1"])
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/app/bert_expansion.py", line 109, in set_expansion_tensorized
    gold_repr_list.append(self.extract_representation(" ".join(seed), x, dim))
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/app/bert_expansion.py", line 317, in extract_representation
    output_all_encoded_layers=output_all_encoded_layers)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 626, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 193, in forward
    words_embeddings = self.word_embeddings(input_ids)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/mnt/castor/seas_home/d/danielkh/ideaProjects/bert-analogies/env3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Here is a summary of what I do in my code:

self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
self.model = BertModel.from_pretrained(bert_model) # loading the model  
self.model.to(self.device) # without this there is no error, but it runs in CPU (instead of GPU). 
self.model.eval() # declaring to the system that we're only doing 'forward' calculations 

# creating the input tensors here 
...
# move the tensors to the target device 
input_ids_tensor.to(self.device)
segment_ids_tensor.to(self.device)
input_mask_tensor.to(self.device)
output_all_encoded_layers.to(self.device)
encoded_layers, _ = self.model(input_ids_tensor, segment_ids_tensor, input_mask_tensor, output_all_encoded_layers=output_all_encoded_layers)

When I don’t have model.to(device) the code works fine, but I think it only uses CPU only. When I add it, it fails with the above error.

I did a little investigation and printed the inputs to .model(.) to see if they are properly copied to device:

print("\n * input_ids_tensor \n ")
print(input_ids_tensor)
print(input_ids_tensor.device)

print("\n * segment_ids_tensor \n ")
print(segment_ids_tensor)
print(segment_ids_tensor.device)

print("\n * input_mask_tensor \n ")
print(input_mask_tensor)
print(input_mask_tensor.device)

print("\n * self.device \n ")
print(self.device)

which outputs:

 * input_ids_tensor  
tensor([[ 101, 5334, 2148, 1035, 3792, 3146,  102, 5334,  102,    0,    0],
        [ 101, 5334, 2148, 1035, 3792, 3146,  102, 2148, 1035, 3792,  102],
        [ 101, 5334, 2148, 1035, 3792, 3146,  102, 3146,  102,    0,    0]])
cpu

 * segment_ids_tensor  
tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
        [0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0]])
cpu

 * input_mask_tensor  
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])
cpu

 * self.device 
cuda:0

As it can be seen, the tensors are still cpu, even after running .to(device).

Any thoughts where things are going wrong?

[kkadowa]

You failed to move the tensors to GPU. Replace your code with this:

input_ids_tensor = input_ids_tensor.to(self.device)
segment_ids_tensor = segment_ids_tensor.to(self.device)
input_mask_tensor = input_mask_tensor.to(self.device)

[danyaljj]

Ah I didn’t realize they don’t work in-place (unlike the syntax for model files model.to(device)).