Attempted to restart training on a COCO dataset after 2 epochs... failed with runtime error

was training COCO… everything went smoothly and I managed to get into 3rd epoch and paused the training. Went to restart, and got errors. Unsure how to proceed with debugging or cleaning up…

CUDA_VISIBLE_DEVICES=0 python trainval_net.py --dataset coco --net res101 --bs 1 --nw 1 --lr .001 --lr_decay_step 10 --cuda --r true --checksession 1 --checkepoch 2 --checkpoint 234531 --use_tfb

234532 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
loading checkpoint models/res101/coco/faster_rcnn_1_2_234531.pth
loaded checkpoint models/res101/coco/faster_rcnn_1_2_234531.pth
Traceback (most recent call last):
  File "trainval_net.py", line 339, in <module>
    optimizer.step()
  File "/python3.6/site-packages/torch/optim/sgd.py", line 101, in step
    buf.mul_(momentum).add_(1 - dampening, d_p)
RuntimeError: expected type torch.FloatTensor but got torch.cuda.FloatTensor
(pytorch1_py36) emcp@k:faster-rcnn.pytorch$ 

[AlexanderHustinx]

I ran into a similar problem. For me that issue was solved when moving the lines:

  if args.cuda:
    fasterRCNN.cuda()

above the assignment of the optimizer, i.e. above:

  if args.optimizer == "adam":
    lr = lr * 0.1
    optimizer = torch.optim.Adam(params)

  elif args.optimizer == "sgd":
    optimizer = torch.optim.SGD(params, momentum=cfg.TRAIN.MOMENTUM)

According to the documentation it is best practice to move the model to GPU prior to initialization/assignment of the optimizer.

[EMCP]

@john2020-210 https://github.com/jwyang/faster-rcnn.pytorch/tree/pytorch-1.0