Error in making prediction on CPU after training the model on GPU

Hi, I trained the model on GPU according to tutorial.

reader = BertQA(bert_model='bert-base-multilingual-cased',
                train_batch_size=256,
                learning_rate=3e-5,
                num_train_epochs=2,
                do_lower_case=False,
                verbose_logging=True,
                output_dir='./temp')

reader.fit(X=(train_examples, train_features))

And before dumping the model, send it to CPU.

reader.model.to('cpu')
reader.device = torch.device('cpu')

But I try to make a prediction on CPU, then following error occurs…

query = 'some sample query...'
prediction = cdqa_pipeline.predict(X=query)

--------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-79-c881b3585457> in <module>
      1 query = ''some sample query...''
----> 2 prediction = cdqa_pipeline.predict(X=query)

~/anaconda3/lib/python3.7/site-packages/cdqa/pipeline/cdqa_sklearn.py in predict(self, X, return_logit)
    158                                                      metadata=self.metadata)
    159             examples, features = self.processor_predict.fit_transform(X=squad_examples)
--> 160             prediction = self.reader.predict((examples, features), return_logit)
    161             return prediction
    162 

~/anaconda3/lib/python3.7/site-packages/cdqa/reader/bertqa_sklearn.py in predict(self, X, return_logit)
   1220             with torch.no_grad():
   1221                 batch_start_logits, batch_end_logits = self.model(
-> 1222                     input_ids, segment_ids, input_mask)
   1223             for i, example_index in enumerate(example_indices):
   1224                 start_logits = batch_start_logits[i].detach().cpu().tolist()

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    144                 raise RuntimeError("module must have its parameters and buffers "
    145                                    "on device {} (device_ids[0]) but found one of "
--> 146                                    "them on device: {}".format(self.src_device_obj, t.device))
    147 
    148         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu

Is there something else I need to do?

[deo4kyo]

Hi, andrelmfarias. I used 8 GPUs(distributed training).

But now I understand what’s wrong. Thanks for your help ndrelmfarias.

[andrelmfarias]

I just tried to train a new model and when print type(model.model) I get

pytorch_pretrained_bert.modeling.BertForQuestionAnswering

Not torch.nn.parallel.data_parallel.DataParallel…

Did you train the model with multiple GPUs with distributed training?

Thanks