Very Slow Inference on CPU

Hi I found the model is very very slow on CPU using the pretrained weight “checkpoint_iter_370000.pth”. I have attached the code below. I have tested different scenarios, and summarize the results below: GPU w pretrained weight: 0.007 sec GPU w/o pretrained weight: 0.007 sec CPU w pretrained weight: 2.829 sec CPU w/o pretrained weight: 0.376 sec

Could you kindly explain why the inference time using CPU and pretrained weight is so slow ?

import argparse
import cv2
import numpy as np
import torch
from models.with_mobilenet import PoseEstimationWithMobileNet
from modules.keypoints import extract_keypoints, group_keypoints
from modules.load_state import load_state
from modules.pose import Pose, propagate_ids
import time
#from val import normalize, pad_width
device = torch.device('cpu')
model_Mobilenet = PoseEstimationWithMobileNet().to(device)
checkpoint = torch.load('checkpoint_iter_370000.pth', map_location=lambda storage, loc: storage)
load_state(model_Mobilenet, checkpoint)
model_Mobilenet.eval()
input = torch.Tensor(2, 3, 368, 368).to(device)

since = time.time()
stages_output= model_Mobilenet(input)
PAF_Mobilenet, Heatmap_Mobilenet = stages_output[-1], stages_output[-2]
print('Mobilenet Inference time is {:2.3f} seconds'.format(time.time() - since))

[xuexingyu24]

Daniil, Yes. It is reproduced with real image. I rounded over the pretrained weight parameters by 4 decimals. The inference speed is back to 0.3 sec. I didn’t find any accuracy drop. For your information. Not sure if you can reproduce this scenario though. Wonder if there is any theoretical explanation on this ?

params = list(model_Mobilenet.parameters())
print('the length of parameters is', len(params))
for i in range(len(params)):
    params[i].data = torch.round(params[i].data*10**4) / 10**4

[Daniil-Osokin]

This can be closed.