Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
RuntimeError: [enforce fail at pybind_state.h:433] . Exception encountered running PythonOp function: ValueError: could not broadcast input array from shape (4) into shape (0)
See original GitHub issueHi,
I have ran this program perfectly several times！ BUT the issue occur. 😦
I can’t fix it by myself.
Please give me a hand.
./start.sh
Found Detectron ops lib: /usr/local/lib/python2.7/dist-packages/torch/lib/libcaffe2_detectron_ops_gpu.so
[E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
INFO train_net.py: 95: Called with args:
INFO train_net.py: 96: Namespace(cfg_file='configs/12_2017_baselines/e2e_mask_rcnn_R-50-FPN_1x.yaml', multi_gpu_testing=False, opts=['OUTPUT_DIR', 'out_dir'], skip_test=False)
INFO train_net.py: 103: cuda version : 10000
INFO train_net.py: 104: cudnn version: 7300
INFO train_net.py: 105: nvidia-smi output:
Sat Nov 10 21:52:10 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.73 Driver Version: 410.73 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN Xp Off | 00000000:01:00.0 On | N/A |
| 23% 25C P8 10W / 250W | 682MiB / 12194MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1076 G /usr/lib/xorg/Xorg 344MiB |
| 0 1928 G compiz 75MiB |
| 0 2489 G ...-token=D729C7639D1670844BD18EC0091B297A 57MiB |
| 0 3496 G ...quest-channel-token=9549767577372815760 192MiB |
+-----------------------------------------------------------------------------+
INFO train_net.py: 106: Training with config:
INFO train_net.py: 107: {'BBOX_XFORM_CLIP': 4.135166556742356,
'CLUSTER': {'ON_CLUSTER': False},
'DATA_LOADER': {'BLOBS_QUEUE_CAPACITY': 8,
'MINIBATCH_QUEUE_SIZE': 64,
'NUM_THREADS': 4},
'DEDUP_BOXES': 0.0625,
'DOWNLOAD_CACHE': u'/tmp/detectron-download-cache',
'EPS': 1e-14,
'EXPECTED_RESULTS': [],
'EXPECTED_RESULTS_ATOL': 0.005,
'EXPECTED_RESULTS_EMAIL': u'',
'EXPECTED_RESULTS_RTOL': 0.1,
'EXPECTED_RESULTS_SIGMA_TOL': 4,
'FAST_RCNN': {'CONV_HEAD_DIM': 256,
'MLP_HEAD_DIM': 1024,
'NUM_STACKED_CONVS': 4,
'ROI_BOX_HEAD': 'fast_rcnn_heads.add_roi_2mlp_head',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 2},
'FPN': {'COARSEST_STRIDE': 32,
'DIM': 256,
'EXTRA_CONV_LEVELS': False,
'FPN_ON': True,
'MULTILEVEL_ROIS': True,
'MULTILEVEL_RPN': True,
'ROI_CANONICAL_LEVEL': 4,
'ROI_CANONICAL_SCALE': 224,
'ROI_MAX_LEVEL': 5,
'ROI_MIN_LEVEL': 2,
'RPN_ANCHOR_START_SIZE': 32,
'RPN_ASPECT_RATIOS': (0.5, 1, 2),
'RPN_MAX_LEVEL': 6,
'RPN_MIN_LEVEL': 2,
'USE_GN': False,
'ZERO_INIT_LATERAL': False},
'GROUP_NORM': {'DIM_PER_GP': -1, 'EPSILON': 1e-05, 'NUM_GROUPS': 32},
'KRCNN': {'CONV_HEAD_DIM': 256,
'CONV_HEAD_KERNEL': 3,
'CONV_INIT': u'GaussianFill',
'DECONV_DIM': 256,
'DECONV_KERNEL': 4,
'DILATION': 1,
'HEATMAP_SIZE': -1,
'INFERENCE_MIN_SIZE': 0,
'KEYPOINT_CONFIDENCE': u'bbox',
'LOSS_WEIGHT': 1.0,
'MIN_KEYPOINT_COUNT_FOR_VALID_MINIBATCH': 20,
'NMS_OKS': False,
'NORMALIZE_BY_VISIBLE_KEYPOINTS': True,
'NUM_KEYPOINTS': -1,
'NUM_STACKED_CONVS': 8,
'ROI_KEYPOINTS_HEAD': u'',
'ROI_XFORM_METHOD': u'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 0,
'UP_SCALE': -1,
'USE_DECONV': False,
'USE_DECONV_OUTPUT': False},
'MATLAB': u'matlab',
'MEMONGER': True,
'MEMONGER_SHARE_ACTIVATIONS': False,
'MODEL': {'BBOX_REG_WEIGHTS': (10.0, 10.0, 5.0, 5.0),
'CLS_AGNOSTIC_BBOX_REG': False,
'CONV_BODY': 'FPN.add_fpn_ResNet50_conv5_body',
'EXECUTION_TYPE': u'dag',
'FASTER_RCNN': True,
'KEYPOINTS_ON': False,
'MASK_ON': True,
'NUM_CLASSES': 4,
'RPN_ONLY': False,
'TYPE': 'generalized_rcnn'},
'MRCNN': {'CLS_SPECIFIC_MASK': True,
'CONV_INIT': 'MSRAFill',
'DILATION': 1,
'DIM_REDUCED': 256,
'RESOLUTION': 28,
'ROI_MASK_HEAD': 'mask_rcnn_heads.mask_rcnn_fcn_head_v1up4convs',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 14,
'ROI_XFORM_SAMPLING_RATIO': 2,
'THRESH_BINARIZE': 0.5,
'UPSAMPLE_RATIO': 1,
'USE_FC_OUTPUT': False,
'WEIGHT_LOSS_MASK': 1.0},
'NUM_GPUS': 1,
'OUTPUT_DIR': 'out_dir',
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'RESNETS': {'NUM_GROUPS': 1,
'RES5_DILATION': 1,
'SHORTCUT_FUNC': u'basic_bn_shortcut',
'STEM_FUNC': u'basic_bn_stem',
'STRIDE_1X1': True,
'TRANS_FUNC': u'bottleneck_transformation',
'WIDTH_PER_GROUP': 64},
'RETINANET': {'ANCHOR_SCALE': 4,
'ASPECT_RATIOS': (0.5, 1.0, 2.0),
'BBOX_REG_BETA': 0.11,
'BBOX_REG_WEIGHT': 1.0,
'CLASS_SPECIFIC_BBOX': False,
'INFERENCE_TH': 0.05,
'LOSS_ALPHA': 0.25,
'LOSS_GAMMA': 2.0,
'NEGATIVE_OVERLAP': 0.4,
'NUM_CONVS': 4,
'POSITIVE_OVERLAP': 0.5,
'PRE_NMS_TOP_N': 1000,
'PRIOR_PROB': 0.01,
'RETINANET_ON': False,
'SCALES_PER_OCTAVE': 3,
'SHARE_CLS_BBOX_TOWER': False,
'SOFTMAX': False},
'RFCN': {'PS_GRID_SIZE': 3},
'RNG_SEED': 3,
'ROOT_DIR': '/home/lenovo/Detectron/detectron',
'RPN': {'ASPECT_RATIOS': (0.5, 1, 2),
'RPN_ON': True,
'SIZES': (64, 128, 256, 512),
'STRIDE': 16},
'SOLVER': {'BASE_LR': 0.0002,
'GAMMA': 0.1,
'LOG_LR_CHANGE_THRESHOLD': 1.1,
'LRS': [],
'LR_POLICY': 'steps_with_decay',
'MAX_ITER': 90000,
'MOMENTUM': 0.9,
'SCALE_MOMENTUM': True,
'SCALE_MOMENTUM_THRESHOLD': 1.1,
'STEPS': [0, 60000, 80000],
'STEP_SIZE': 30000,
'WARM_UP_FACTOR': 0.3333333333333333,
'WARM_UP_ITERS': 500,
'WARM_UP_METHOD': u'linear',
'WEIGHT_DECAY': 0.0001,
'WEIGHT_DECAY_GN': 0.0},
'TEST': {'BBOX_AUG': {'AREA_TH_HI': 32400,
'AREA_TH_LO': 2500,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'COORD_HEUR': u'UNION',
'ENABLED': False,
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False,
'SCORE_HEUR': u'UNION'},
'BBOX_REG': True,
'BBOX_VOTE': {'ENABLED': False,
'SCORING_METHOD': u'ID',
'SCORING_METHOD_BETA': 1.0,
'VOTE_TH': 0.8},
'COMPETITION_MODE': True,
'DATASETS': ('labelme_val',),
'DETECTIONS_PER_IM': 100,
'FORCE_JSON_DATASET_EVAL': True,
'KPS_AUG': {'AREA_TH': 32400,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'ENABLED': False,
'HEUR': u'HM_AVG',
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False},
'MASK_AUG': {'AREA_TH': 32400,
'ASPECT_RATIOS': (),
'ASPECT_RATIO_H_FLIP': False,
'ENABLED': False,
'HEUR': u'SOFT_AVG',
'H_FLIP': False,
'MAX_SIZE': 4000,
'SCALES': (),
'SCALE_H_FLIP': False,
'SCALE_SIZE_DEP': False},
'MAX_SIZE': 1024,
'NMS': 0.5,
'PRECOMPUTED_PROPOSALS': False,
'PROPOSAL_FILES': (),
'PROPOSAL_LIMIT': 2000,
'RPN_MIN_SIZE': 0,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 1000,
'RPN_PRE_NMS_TOP_N': 1000,
'SCALE': 300,
'SCORE_THRESH': 0.05,
'SOFT_NMS': {'ENABLED': False, 'METHOD': u'linear', 'SIGMA': 0.5},
'WEIGHTS': u''},
'TRAIN': {'ASPECT_GROUPING': True,
'AUTO_RESUME': True,
'BATCH_SIZE_PER_IM': 128,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'COPY_WEIGHTS': False,
'CROWD_FILTER_THRESH': 0.7,
'DATASETS': ('labelme_train',),
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'FREEZE_AT': 2,
'FREEZE_CONV_BODY': False,
'GT_MIN_AREA': -1,
'IMS_PER_BATCH': 2,
'MAX_SIZE': 1024,
'PROPOSAL_FILES': (),
'RPN_BATCH_SIZE_PER_IM': 256,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 0,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 2000,
'RPN_STRADDLE_THRESH': 0,
'SCALES': (325,),
'SNAPSHOT_ITERS': 20000,
'USE_FLIPPED': True,
'WEIGHTS': u'/tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-50.pkl'},
'USE_NCCL': False,
'VIS': False,
'VIS_TH': 0.9}
INFO train.py: 144: Building model: generalized_rcnn
WARNING cnn.py: 25: [====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information.
WARNING memonger.py: 55: NOTE: Executing memonger to optimize gradient memory
[I memonger.cc:236] Remapping 93 using 21 shared blobs.
INFO memonger.py: 97: Memonger memory optimization took 0.0118179321289 secs
[I context_gpu.cu:317] GPU 0: 153 MB
[I context_gpu.cu:321] Total: 153 MB
[I context_gpu.cu:317] GPU 0: 320 MB
[I context_gpu.cu:321] Total: 320 MB
INFO train.py: 192: Loading dataset: ('labelme_train',)
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
INFO roidb.py: 49: Appending horizontally-flipped training examples...
INFO roidb.py: 51: Loaded dataset: labelme_train
INFO roidb.py: 135: Filtered 0 roidb entries: 562 -> 562
INFO roidb.py: 67: Computing bounding-box regression targets...
INFO roidb.py: 69: done
INFO train.py: 196: 562 roidb entries
INFO net.py: 60: Loading weights from: /tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-50.pkl
INFO net.py: 96: conv1_w loaded from weights file into gpu_0/conv1_w: (64, 3, 7, 7)
INFO net.py: 96: res_conv1_bn_s loaded from weights file into gpu_0/res_conv1_bn_s: (64,)
INFO net.py: 96: res_conv1_bn_b loaded from weights file into gpu_0/res_conv1_bn_b: (64,)
INFO net.py: 96: res2_0_branch2a_w loaded from weights file into gpu_0/res2_0_branch2a_w: (64, 64, 1, 1)
INFO net.py: 96: res2_0_branch2a_bn_s loaded from weights file into gpu_0/res2_0_branch2a_bn_s: (64,)
INFO net.py: 96: res2_0_branch2a_bn_b loaded from weights file into gpu_0/res2_0_branch2a_bn_b: (64,)
INFO net.py: 96: res2_0_branch2b_w loaded from weights file into gpu_0/res2_0_branch2b_w: (64, 64, 3, 3)
INFO net.py: 96: res2_0_branch2b_bn_s loaded from weights file into gpu_0/res2_0_branch2b_bn_s: (64,)
INFO net.py: 96: res2_0_branch2b_bn_b loaded from weights file into gpu_0/res2_0_branch2b_bn_b: (64,)
INFO net.py: 96: res2_0_branch2c_w loaded from weights file into gpu_0/res2_0_branch2c_w: (256, 64, 1, 1)
INFO net.py: 96: res2_0_branch2c_bn_s loaded from weights file into gpu_0/res2_0_branch2c_bn_s: (256,)
INFO net.py: 96: res2_0_branch2c_bn_b loaded from weights file into gpu_0/res2_0_branch2c_bn_b: (256,)
INFO net.py: 96: res2_0_branch1_w loaded from weights file into gpu_0/res2_0_branch1_w: (256, 64, 1, 1)
INFO net.py: 96: res2_0_branch1_bn_s loaded from weights file into gpu_0/res2_0_branch1_bn_s: (256,)
INFO net.py: 96: res2_0_branch1_bn_b loaded from weights file into gpu_0/res2_0_branch1_bn_b: (256,)
INFO net.py: 96: res2_1_branch2a_w loaded from weights file into gpu_0/res2_1_branch2a_w: (64, 256, 1, 1)
INFO net.py: 96: res2_1_branch2a_bn_s loaded from weights file into gpu_0/res2_1_branch2a_bn_s: (64,)
INFO net.py: 96: res2_1_branch2a_bn_b loaded from weights file into gpu_0/res2_1_branch2a_bn_b: (64,)
INFO net.py: 96: res2_1_branch2b_w loaded from weights file into gpu_0/res2_1_branch2b_w: (64, 64, 3, 3)
INFO net.py: 96: res2_1_branch2b_bn_s loaded from weights file into gpu_0/res2_1_branch2b_bn_s: (64,)
INFO net.py: 96: res2_1_branch2b_bn_b loaded from weights file into gpu_0/res2_1_branch2b_bn_b: (64,)
INFO net.py: 96: res2_1_branch2c_w loaded from weights file into gpu_0/res2_1_branch2c_w: (256, 64, 1, 1)
INFO net.py: 96: res2_1_branch2c_bn_s loaded from weights file into gpu_0/res2_1_branch2c_bn_s: (256,)
INFO net.py: 96: res2_1_branch2c_bn_b loaded from weights file into gpu_0/res2_1_branch2c_bn_b: (256,)
INFO net.py: 96: res2_2_branch2a_w loaded from weights file into gpu_0/res2_2_branch2a_w: (64, 256, 1, 1)
INFO net.py: 96: res2_2_branch2a_bn_s loaded from weights file into gpu_0/res2_2_branch2a_bn_s: (64,)
INFO net.py: 96: res2_2_branch2a_bn_b loaded from weights file into gpu_0/res2_2_branch2a_bn_b: (64,)
INFO net.py: 96: res2_2_branch2b_w loaded from weights file into gpu_0/res2_2_branch2b_w: (64, 64, 3, 3)
INFO net.py: 96: res2_2_branch2b_bn_s loaded from weights file into gpu_0/res2_2_branch2b_bn_s: (64,)
INFO net.py: 96: res2_2_branch2b_bn_b loaded from weights file into gpu_0/res2_2_branch2b_bn_b: (64,)
INFO net.py: 96: res2_2_branch2c_w loaded from weights file into gpu_0/res2_2_branch2c_w: (256, 64, 1, 1)
INFO net.py: 96: res2_2_branch2c_bn_s loaded from weights file into gpu_0/res2_2_branch2c_bn_s: (256,)
INFO net.py: 96: res2_2_branch2c_bn_b loaded from weights file into gpu_0/res2_2_branch2c_bn_b: (256,)
INFO net.py: 96: res3_0_branch2a_w loaded from weights file into gpu_0/res3_0_branch2a_w: (128, 256, 1, 1)
INFO net.py: 96: res3_0_branch2a_bn_s loaded from weights file into gpu_0/res3_0_branch2a_bn_s: (128,)
INFO net.py: 96: res3_0_branch2a_bn_b loaded from weights file into gpu_0/res3_0_branch2a_bn_b: (128,)
INFO net.py: 96: res3_0_branch2b_w loaded from weights file into gpu_0/res3_0_branch2b_w: (128, 128, 3, 3)
INFO net.py: 96: res3_0_branch2b_bn_s loaded from weights file into gpu_0/res3_0_branch2b_bn_s: (128,)
INFO net.py: 96: res3_0_branch2b_bn_b loaded from weights file into gpu_0/res3_0_branch2b_bn_b: (128,)
INFO net.py: 96: res3_0_branch2c_w loaded from weights file into gpu_0/res3_0_branch2c_w: (512, 128, 1, 1)
INFO net.py: 96: res3_0_branch2c_bn_s loaded from weights file into gpu_0/res3_0_branch2c_bn_s: (512,)
INFO net.py: 96: res3_0_branch2c_bn_b loaded from weights file into gpu_0/res3_0_branch2c_bn_b: (512,)
INFO net.py: 96: res3_0_branch1_w loaded from weights file into gpu_0/res3_0_branch1_w: (512, 256, 1, 1)
INFO net.py: 96: res3_0_branch1_bn_s loaded from weights file into gpu_0/res3_0_branch1_bn_s: (512,)
INFO net.py: 96: res3_0_branch1_bn_b loaded from weights file into gpu_0/res3_0_branch1_bn_b: (512,)
INFO net.py: 96: res3_1_branch2a_w loaded from weights file into gpu_0/res3_1_branch2a_w: (128, 512, 1, 1)
INFO net.py: 96: res3_1_branch2a_bn_s loaded from weights file into gpu_0/res3_1_branch2a_bn_s: (128,)
INFO net.py: 96: res3_1_branch2a_bn_b loaded from weights file into gpu_0/res3_1_branch2a_bn_b: (128,)
INFO net.py: 96: res3_1_branch2b_w loaded from weights file into gpu_0/res3_1_branch2b_w: (128, 128, 3, 3)
INFO net.py: 96: res3_1_branch2b_bn_s loaded from weights file into gpu_0/res3_1_branch2b_bn_s: (128,)
INFO net.py: 96: res3_1_branch2b_bn_b loaded from weights file into gpu_0/res3_1_branch2b_bn_b: (128,)
INFO net.py: 96: res3_1_branch2c_w loaded from weights file into gpu_0/res3_1_branch2c_w: (512, 128, 1, 1)
INFO net.py: 96: res3_1_branch2c_bn_s loaded from weights file into gpu_0/res3_1_branch2c_bn_s: (512,)
INFO net.py: 96: res3_1_branch2c_bn_b loaded from weights file into gpu_0/res3_1_branch2c_bn_b: (512,)
INFO net.py: 96: res3_2_branch2a_w loaded from weights file into gpu_0/res3_2_branch2a_w: (128, 512, 1, 1)
INFO net.py: 96: res3_2_branch2a_bn_s loaded from weights file into gpu_0/res3_2_branch2a_bn_s: (128,)
INFO net.py: 96: res3_2_branch2a_bn_b loaded from weights file into gpu_0/res3_2_branch2a_bn_b: (128,)
INFO net.py: 96: res3_2_branch2b_w loaded from weights file into gpu_0/res3_2_branch2b_w: (128, 128, 3, 3)
INFO net.py: 96: res3_2_branch2b_bn_s loaded from weights file into gpu_0/res3_2_branch2b_bn_s: (128,)
INFO net.py: 96: res3_2_branch2b_bn_b loaded from weights file into gpu_0/res3_2_branch2b_bn_b: (128,)
INFO net.py: 96: res3_2_branch2c_w loaded from weights file into gpu_0/res3_2_branch2c_w: (512, 128, 1, 1)
INFO net.py: 96: res3_2_branch2c_bn_s loaded from weights file into gpu_0/res3_2_branch2c_bn_s: (512,)
INFO net.py: 96: res3_2_branch2c_bn_b loaded from weights file into gpu_0/res3_2_branch2c_bn_b: (512,)
INFO net.py: 96: res3_3_branch2a_w loaded from weights file into gpu_0/res3_3_branch2a_w: (128, 512, 1, 1)
INFO net.py: 96: res3_3_branch2a_bn_s loaded from weights file into gpu_0/res3_3_branch2a_bn_s: (128,)
INFO net.py: 96: res3_3_branch2a_bn_b loaded from weights file into gpu_0/res3_3_branch2a_bn_b: (128,)
INFO net.py: 96: res3_3_branch2b_w loaded from weights file into gpu_0/res3_3_branch2b_w: (128, 128, 3, 3)
INFO net.py: 96: res3_3_branch2b_bn_s loaded from weights file into gpu_0/res3_3_branch2b_bn_s: (128,)
INFO net.py: 96: res3_3_branch2b_bn_b loaded from weights file into gpu_0/res3_3_branch2b_bn_b: (128,)
INFO net.py: 96: res3_3_branch2c_w loaded from weights file into gpu_0/res3_3_branch2c_w: (512, 128, 1, 1)
INFO net.py: 96: res3_3_branch2c_bn_s loaded from weights file into gpu_0/res3_3_branch2c_bn_s: (512,)
INFO net.py: 96: res3_3_branch2c_bn_b loaded from weights file into gpu_0/res3_3_branch2c_bn_b: (512,)
INFO net.py: 96: res4_0_branch2a_w loaded from weights file into gpu_0/res4_0_branch2a_w: (256, 512, 1, 1)
INFO net.py: 96: res4_0_branch2a_bn_s loaded from weights file into gpu_0/res4_0_branch2a_bn_s: (256,)
INFO net.py: 96: res4_0_branch2a_bn_b loaded from weights file into gpu_0/res4_0_branch2a_bn_b: (256,)
INFO net.py: 96: res4_0_branch2b_w loaded from weights file into gpu_0/res4_0_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_0_branch2b_bn_s loaded from weights file into gpu_0/res4_0_branch2b_bn_s: (256,)
INFO net.py: 96: res4_0_branch2b_bn_b loaded from weights file into gpu_0/res4_0_branch2b_bn_b: (256,)
INFO net.py: 96: res4_0_branch2c_w loaded from weights file into gpu_0/res4_0_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_0_branch2c_bn_s loaded from weights file into gpu_0/res4_0_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_0_branch2c_bn_b loaded from weights file into gpu_0/res4_0_branch2c_bn_b: (1024,)
INFO net.py: 96: res4_0_branch1_w loaded from weights file into gpu_0/res4_0_branch1_w: (1024, 512, 1, 1)
INFO net.py: 96: res4_0_branch1_bn_s loaded from weights file into gpu_0/res4_0_branch1_bn_s: (1024,)
INFO net.py: 96: res4_0_branch1_bn_b loaded from weights file into gpu_0/res4_0_branch1_bn_b: (1024,)
INFO net.py: 96: res4_1_branch2a_w loaded from weights file into gpu_0/res4_1_branch2a_w: (256, 1024, 1, 1)
INFO net.py: 96: res4_1_branch2a_bn_s loaded from weights file into gpu_0/res4_1_branch2a_bn_s: (256,)
INFO net.py: 96: res4_1_branch2a_bn_b loaded from weights file into gpu_0/res4_1_branch2a_bn_b: (256,)
INFO net.py: 96: res4_1_branch2b_w loaded from weights file into gpu_0/res4_1_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_1_branch2b_bn_s loaded from weights file into gpu_0/res4_1_branch2b_bn_s: (256,)
INFO net.py: 96: res4_1_branch2b_bn_b loaded from weights file into gpu_0/res4_1_branch2b_bn_b: (256,)
INFO net.py: 96: res4_1_branch2c_w loaded from weights file into gpu_0/res4_1_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_1_branch2c_bn_s loaded from weights file into gpu_0/res4_1_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_1_branch2c_bn_b loaded from weights file into gpu_0/res4_1_branch2c_bn_b: (1024,)
INFO net.py: 96: res4_2_branch2a_w loaded from weights file into gpu_0/res4_2_branch2a_w: (256, 1024, 1, 1)
INFO net.py: 96: res4_2_branch2a_bn_s loaded from weights file into gpu_0/res4_2_branch2a_bn_s: (256,)
INFO net.py: 96: res4_2_branch2a_bn_b loaded from weights file into gpu_0/res4_2_branch2a_bn_b: (256,)
INFO net.py: 96: res4_2_branch2b_w loaded from weights file into gpu_0/res4_2_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_2_branch2b_bn_s loaded from weights file into gpu_0/res4_2_branch2b_bn_s: (256,)
INFO net.py: 96: res4_2_branch2b_bn_b loaded from weights file into gpu_0/res4_2_branch2b_bn_b: (256,)
INFO net.py: 96: res4_2_branch2c_w loaded from weights file into gpu_0/res4_2_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_2_branch2c_bn_s loaded from weights file into gpu_0/res4_2_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_2_branch2c_bn_b loaded from weights file into gpu_0/res4_2_branch2c_bn_b: (1024,)
INFO net.py: 96: res4_3_branch2a_w loaded from weights file into gpu_0/res4_3_branch2a_w: (256, 1024, 1, 1)
INFO net.py: 96: res4_3_branch2a_bn_s loaded from weights file into gpu_0/res4_3_branch2a_bn_s: (256,)
INFO net.py: 96: res4_3_branch2a_bn_b loaded from weights file into gpu_0/res4_3_branch2a_bn_b: (256,)
INFO net.py: 96: res4_3_branch2b_w loaded from weights file into gpu_0/res4_3_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_3_branch2b_bn_s loaded from weights file into gpu_0/res4_3_branch2b_bn_s: (256,)
INFO net.py: 96: res4_3_branch2b_bn_b loaded from weights file into gpu_0/res4_3_branch2b_bn_b: (256,)
INFO net.py: 96: res4_3_branch2c_w loaded from weights file into gpu_0/res4_3_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_3_branch2c_bn_s loaded from weights file into gpu_0/res4_3_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_3_branch2c_bn_b loaded from weights file into gpu_0/res4_3_branch2c_bn_b: (1024,)
INFO net.py: 96: res4_4_branch2a_w loaded from weights file into gpu_0/res4_4_branch2a_w: (256, 1024, 1, 1)
INFO net.py: 96: res4_4_branch2a_bn_s loaded from weights file into gpu_0/res4_4_branch2a_bn_s: (256,)
INFO net.py: 96: res4_4_branch2a_bn_b loaded from weights file into gpu_0/res4_4_branch2a_bn_b: (256,)
INFO net.py: 96: res4_4_branch2b_w loaded from weights file into gpu_0/res4_4_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_4_branch2b_bn_s loaded from weights file into gpu_0/res4_4_branch2b_bn_s: (256,)
INFO net.py: 96: res4_4_branch2b_bn_b loaded from weights file into gpu_0/res4_4_branch2b_bn_b: (256,)
INFO net.py: 96: res4_4_branch2c_w loaded from weights file into gpu_0/res4_4_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_4_branch2c_bn_s loaded from weights file into gpu_0/res4_4_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_4_branch2c_bn_b loaded from weights file into gpu_0/res4_4_branch2c_bn_b: (1024,)
INFO net.py: 96: res4_5_branch2a_w loaded from weights file into gpu_0/res4_5_branch2a_w: (256, 1024, 1, 1)
INFO net.py: 96: res4_5_branch2a_bn_s loaded from weights file into gpu_0/res4_5_branch2a_bn_s: (256,)
INFO net.py: 96: res4_5_branch2a_bn_b loaded from weights file into gpu_0/res4_5_branch2a_bn_b: (256,)
INFO net.py: 96: res4_5_branch2b_w loaded from weights file into gpu_0/res4_5_branch2b_w: (256, 256, 3, 3)
INFO net.py: 96: res4_5_branch2b_bn_s loaded from weights file into gpu_0/res4_5_branch2b_bn_s: (256,)
INFO net.py: 96: res4_5_branch2b_bn_b loaded from weights file into gpu_0/res4_5_branch2b_bn_b: (256,)
INFO net.py: 96: res4_5_branch2c_w loaded from weights file into gpu_0/res4_5_branch2c_w: (1024, 256, 1, 1)
INFO net.py: 96: res4_5_branch2c_bn_s loaded from weights file into gpu_0/res4_5_branch2c_bn_s: (1024,)
INFO net.py: 96: res4_5_branch2c_bn_b loaded from weights file into gpu_0/res4_5_branch2c_bn_b: (1024,)
INFO net.py: 96: res5_0_branch2a_w loaded from weights file into gpu_0/res5_0_branch2a_w: (512, 1024, 1, 1)
INFO net.py: 96: res5_0_branch2a_bn_s loaded from weights file into gpu_0/res5_0_branch2a_bn_s: (512,)
INFO net.py: 96: res5_0_branch2a_bn_b loaded from weights file into gpu_0/res5_0_branch2a_bn_b: (512,)
INFO net.py: 96: res5_0_branch2b_w loaded from weights file into gpu_0/res5_0_branch2b_w: (512, 512, 3, 3)
INFO net.py: 96: res5_0_branch2b_bn_s loaded from weights file into gpu_0/res5_0_branch2b_bn_s: (512,)
INFO net.py: 96: res5_0_branch2b_bn_b loaded from weights file into gpu_0/res5_0_branch2b_bn_b: (512,)
INFO net.py: 96: res5_0_branch2c_w loaded from weights file into gpu_0/res5_0_branch2c_w: (2048, 512, 1, 1)
INFO net.py: 96: res5_0_branch2c_bn_s loaded from weights file into gpu_0/res5_0_branch2c_bn_s: (2048,)
INFO net.py: 96: res5_0_branch2c_bn_b loaded from weights file into gpu_0/res5_0_branch2c_bn_b: (2048,)
INFO net.py: 96: res5_0_branch1_w loaded from weights file into gpu_0/res5_0_branch1_w: (2048, 1024, 1, 1)
INFO net.py: 96: res5_0_branch1_bn_s loaded from weights file into gpu_0/res5_0_branch1_bn_s: (2048,)
INFO net.py: 96: res5_0_branch1_bn_b loaded from weights file into gpu_0/res5_0_branch1_bn_b: (2048,)
INFO net.py: 96: res5_1_branch2a_w loaded from weights file into gpu_0/res5_1_branch2a_w: (512, 2048, 1, 1)
INFO net.py: 96: res5_1_branch2a_bn_s loaded from weights file into gpu_0/res5_1_branch2a_bn_s: (512,)
INFO net.py: 96: res5_1_branch2a_bn_b loaded from weights file into gpu_0/res5_1_branch2a_bn_b: (512,)
INFO net.py: 96: res5_1_branch2b_w loaded from weights file into gpu_0/res5_1_branch2b_w: (512, 512, 3, 3)
INFO net.py: 96: res5_1_branch2b_bn_s loaded from weights file into gpu_0/res5_1_branch2b_bn_s: (512,)
INFO net.py: 96: res5_1_branch2b_bn_b loaded from weights file into gpu_0/res5_1_branch2b_bn_b: (512,)
INFO net.py: 96: res5_1_branch2c_w loaded from weights file into gpu_0/res5_1_branch2c_w: (2048, 512, 1, 1)
INFO net.py: 96: res5_1_branch2c_bn_s loaded from weights file into gpu_0/res5_1_branch2c_bn_s: (2048,)
INFO net.py: 96: res5_1_branch2c_bn_b loaded from weights file into gpu_0/res5_1_branch2c_bn_b: (2048,)
INFO net.py: 96: res5_2_branch2a_w loaded from weights file into gpu_0/res5_2_branch2a_w: (512, 2048, 1, 1)
INFO net.py: 96: res5_2_branch2a_bn_s loaded from weights file into gpu_0/res5_2_branch2a_bn_s: (512,)
INFO net.py: 96: res5_2_branch2a_bn_b loaded from weights file into gpu_0/res5_2_branch2a_bn_b: (512,)
INFO net.py: 96: res5_2_branch2b_w loaded from weights file into gpu_0/res5_2_branch2b_w: (512, 512, 3, 3)
INFO net.py: 96: res5_2_branch2b_bn_s loaded from weights file into gpu_0/res5_2_branch2b_bn_s: (512,)
INFO net.py: 96: res5_2_branch2b_bn_b loaded from weights file into gpu_0/res5_2_branch2b_bn_b: (512,)
INFO net.py: 96: res5_2_branch2c_w loaded from weights file into gpu_0/res5_2_branch2c_w: (2048, 512, 1, 1)
INFO net.py: 96: res5_2_branch2c_bn_s loaded from weights file into gpu_0/res5_2_branch2c_bn_s: (2048,)
INFO net.py: 96: res5_2_branch2c_bn_b loaded from weights file into gpu_0/res5_2_branch2c_bn_b: (2048,)
INFO net.py: 89: fpn_inner_res5_2_sum_w not found
INFO net.py: 89: fpn_inner_res5_2_sum_b not found
INFO net.py: 89: fpn_inner_res4_5_sum_lateral_w not found
INFO net.py: 89: fpn_inner_res4_5_sum_lateral_b not found
INFO net.py: 89: fpn_inner_res3_3_sum_lateral_w not found
INFO net.py: 89: fpn_inner_res3_3_sum_lateral_b not found
INFO net.py: 89: fpn_inner_res2_2_sum_lateral_w not found
INFO net.py: 89: fpn_inner_res2_2_sum_lateral_b not found
INFO net.py: 89: fpn_res5_2_sum_w not found
INFO net.py: 89: fpn_res5_2_sum_b not found
INFO net.py: 89: fpn_res4_5_sum_w not found
INFO net.py: 89: fpn_res4_5_sum_b not found
INFO net.py: 89: fpn_res3_3_sum_w not found
INFO net.py: 89: fpn_res3_3_sum_b not found
INFO net.py: 89: fpn_res2_2_sum_w not found
INFO net.py: 89: fpn_res2_2_sum_b not found
INFO net.py: 89: conv_rpn_fpn2_w not found
INFO net.py: 89: conv_rpn_fpn2_b not found
INFO net.py: 89: rpn_cls_logits_fpn2_w not found
INFO net.py: 89: rpn_cls_logits_fpn2_b not found
INFO net.py: 89: rpn_bbox_pred_fpn2_w not found
INFO net.py: 89: rpn_bbox_pred_fpn2_b not found
INFO net.py: 89: fc6_w not found
INFO net.py: 89: fc6_b not found
INFO net.py: 89: fc7_w not found
INFO net.py: 89: fc7_b not found
INFO net.py: 89: cls_score_w not found
INFO net.py: 89: cls_score_b not found
INFO net.py: 89: bbox_pred_w not found
INFO net.py: 89: bbox_pred_b not found
INFO net.py: 89: fcn1_w not found
INFO net.py: 89: fcn1_b not found
INFO net.py: 89: fcn2_w not found
INFO net.py: 89: fcn2_b not found
INFO net.py: 89: fcn3_w not found
INFO net.py: 89: fcn3_b not found
INFO net.py: 89: fcn4_w not found
INFO net.py: 89: fcn4_b not found
INFO net.py: 89: conv5_mask_w not found
INFO net.py: 89: conv5_mask_b not found
INFO net.py: 89: mask_fcn_logits_w not found
INFO net.py: 89: mask_fcn_logits_b not found
INFO net.py: 133: res2_1_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res3_1_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_2_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_0_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res2_2_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res3_3_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_1_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res3_3_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_4_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_5_branch2b_b preserved in workspace (unused)
INFO net.py: 133: conv1_b preserved in workspace (unused)
INFO net.py: 133: fc1000_b preserved in workspace (unused)
INFO net.py: 133: fc1000_w preserved in workspace (unused)
INFO net.py: 133: res3_2_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res3_2_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res2_0_branch1_b preserved in workspace (unused)
INFO net.py: 133: res4_2_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res2_1_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res5_0_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_5_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res4_1_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_3_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_0_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_2_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res2_0_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_0_branch1_b preserved in workspace (unused)
INFO net.py: 133: res2_2_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res3_2_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res3_0_branch1_b preserved in workspace (unused)
INFO net.py: 133: res3_1_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res2_0_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res2_1_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res4_1_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_0_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res4_1_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res2_2_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res5_2_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_5_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res3_0_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res3_1_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_1_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_1_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_4_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_2_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res3_3_branch2b_b preserved in workspace (unused)
INFO net.py: 133: res4_4_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res4_3_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_0_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res5_2_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_0_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res3_0_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res5_0_branch1_b preserved in workspace (unused)
INFO net.py: 133: res3_0_branch2c_b preserved in workspace (unused)
INFO net.py: 133: res2_0_branch2a_b preserved in workspace (unused)
INFO net.py: 133: res4_3_branch2c_b preserved in workspace (unused)
[I net_dag_utils.cc:102] Operator graph pruning prior to chain compute took: 0.00035461 secs
INFO train.py: 180: Outputs saved to: /home/lenovo/Detectron/detectron/out_dir/train/labelme_train/generalized_rcnn
INFO loader.py: 230: Pre-filling mini-batch queue...
INFO loader.py: 235: [0/64]
INFO loader.py: 235: [0/64]
INFO loader.py: 235: [2/64]
[I context_gpu.cu:317] GPU 0: 448 MB
[I context_gpu.cu:321] Total: 448 MB
INFO loader.py: 235: [2/64]
INFO loader.py: 235: [0/64]
[I context_gpu.cu:317] GPU 0: 581 MB
[I context_gpu.cu:321] Total: 581 MB
INFO loader.py: 235: [3/64]
INFO loader.py: 235: [7/64]
INFO loader.py: 235: [11/64]
INFO loader.py: 235: [15/64]
INFO loader.py: 235: [19/64]
INFO loader.py: 235: [23/64]
INFO loader.py: 235: [27/64]
INFO loader.py: 235: [31/64]
INFO loader.py: 235: [35/64]
INFO loader.py: 235: [39/64]
INFO loader.py: 235: [42/64]
INFO loader.py: 235: [46/64]
INFO loader.py: 235: [50/64]
INFO loader.py: 235: [54/64]
INFO loader.py: 235: [58/64]
INFO loader.py: 235: [62/64]
INFO detector.py: 479: Changing learning rate 0.000000 -> 0.000067 at iter 0
[I net_async_base.h:196] Using specified CPU pool size: 4; device id: -1
[I net_async_base.h:201] Created new CPU pool, size: 4; device id: -1
[I context_gpu.cu:317] GPU 0: 710 MB
[I context_gpu.cu:321] Total: 710 MB
[I context_gpu.cu:317] GPU 0: 851 MB
[I context_gpu.cu:321] Total: 851 MB
[I context_gpu.cu:317] GPU 0: 979 MB
[I context_gpu.cu:321] Total: 979 MB
[I context_gpu.cu:317] GPU 0: 1112 MB
[I context_gpu.cu:321] Total: 1112 MB
[E pybind_state.h:432] Exception encountered running PythonOp function: ValueError: could not broadcast input array from shape (4) into shape (0)
At:
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(233): _expand_bbox_targets
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(174): _sample_rois
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(112): add_fast_rcnn_blobs
/home/lenovo/Detectron/detectron/detectron/ops/collect_and_distribute_fpn_rpn_proposals.py(62): forward
[E net_async_base.cc:373] [enforce fail at pybind_state.h:433] . Exception encountered running PythonOp function: ValueError: could not broadcast input array from shape (4) into shape (0)
At:
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(233): _expand_bbox_targets
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(174): _sample_rois
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(112): add_fast_rcnn_blobs
/home/lenovo/Detectron/detectron/detectron/ops/collect_and_distribute_fpn_rpn_proposals.py(62): forward
Error from operator:
input: "gpu_0/rpn_rois_fpn2" input: "gpu_0/rpn_rois_fpn3" input: "gpu_0/rpn_rois_fpn4" input: "gpu_0/rpn_rois_fpn5" input: "gpu_0/rpn_rois_fpn6" input: "gpu_0/rpn_roi_probs_fpn2" input: "gpu_0/rpn_roi_probs_fpn3" input: "gpu_0/rpn_roi_probs_fpn4" input: "gpu_0/rpn_roi_probs_fpn5" input: "gpu_0/rpn_roi_probs_fpn6" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois" output: "gpu_0/labels_int32" output: "gpu_0/bbox_targets" output: "gpu_0/bbox_inside_weights" output: "gpu_0/bbox_outside_weights" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_fpn2" output: "gpu_0/rois_fpn3" output: "gpu_0/rois_fpn4" output: "gpu_0/rois_fpn5" output: "gpu_0/rois_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "CollectAndDistributeFpnRpnProposalsOp:gpu_0/rpn_rois_fpn2,gpu_0/rpn_rois_fpn3,gpu_0/rpn_rois_fpn4,gpu_0/rpn_rois_fpn5,gpu_0/rpn_rois_fpn6,gpu_0/rpn_roi_probs_fpn2,gpu_0/rpn_roi_probs_fpn3,gpu_0/rpn_roi_probs_fpn4,gpu_0/rpn_roi_probs_fpn5,gpu_0/rpn_roi_probs_fpn6,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:5" } arg { name: "grad_output_indices" } device_option { device_type: 0 }Error from operator:
input: "gpu_0/rpn_rois_fpn2" input: "gpu_0/rpn_rois_fpn3" input: "gpu_0/rpn_rois_fpn4" input: "gpu_0/rpn_rois_fpn5" input: "gpu_0/rpn_rois_fpn6" input: "gpu_0/rpn_roi_probs_fpn2" input: "gpu_0/rpn_roi_probs_fpn3" input: "gpu_0/rpn_roi_probs_fpn4" input: "gpu_0/rpn_roi_probs_fpn5" input: "gpu_0/rpn_roi_probs_fpn6" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois" output: "gpu_0/labels_int32" output: "gpu_0/bbox_targets" output: "gpu_0/bbox_inside_weights" output: "gpu_0/bbox_outside_weights" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_fpn2" output: "gpu_0/rois_fpn3" output: "gpu_0/rois_fpn4" output: "gpu_0/rois_fpn5" output: "gpu_0/rois_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "CollectAndDistributeFpnRpnProposalsOp:gpu_0/rpn_rois_fpn2,gpu_0/rpn_rois_fpn3,gpu_0/rpn_rois_fpn4,gpu_0/rpn_rois_fpn5,gpu_0/rpn_rois_fpn6,gpu_0/rpn_roi_probs_fpn2,gpu_0/rpn_roi_probs_fpn3,gpu_0/rpn_roi_probs_fpn4,gpu_0/rpn_roi_probs_fpn5,gpu_0/rpn_roi_probs_fpn6,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:5" } arg { name: "grad_output_indices" } device_option { device_type: 1 device_id: 0 }frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*) + 0x76 (0x7f68b8086b46 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: <unknown function> + 0xae9c7 (0x7f68cb3599c7 in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #2: <unknown function> + 0xabbab (0x7f68cb356bab in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #3: <unknown function> + 0xeed4f (0x7f68cb399d4f in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #4: <unknown function> + 0xee27a (0x7f68cb39927a in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #5: caffe2::AsyncNetBase::run(int, int) + 0x154 (0x7f68ca0dc544 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #6: <unknown function> + 0x1311845 (0x7f68ca0e4845 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #7: <unknown function> + 0x130c18b (0x7f68ca0df18b in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #8: <unknown function> + 0xb8c80 (0x7f68f9404c80 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #9: <unknown function> + 0x76ba (0x7f68ffbbd6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #10: clone + 0x6d (0x7f68ff8f341d in /lib/x86_64-linux-gnu/libc.so.6)
, op Python
WARNING workspace.py: 187: Original python traceback for operator `219` in network `generalized_rcnn` in exception above (most recent call last):
WARNING workspace.py: 192: File "tools/train_net.py", line 132, in <module>
WARNING workspace.py: 192: File "tools/train_net.py", line 114, in main
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/utils/train.py", line 53, in train_model
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/utils/train.py", line 145, in create_model
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/model_builder.py", line 124, in create
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/model_builder.py", line 89, in generalized_rcnn
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/model_builder.py", line 229, in build_generic_detection_model
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/optimizer.py", line 40, in build_data_parallel_model
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/optimizer.py", line 63, in _build_forward_graph
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/model_builder.py", line 189, in _single_gpu_build_func
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/rpn_heads.py", line 46, in add_generic_rpn_outputs
WARNING workspace.py: 192: File "/home/lenovo/Detectron/detectron/detectron/modeling/FPN.py", line 449, in add_fpn_rpn_losses
Traceback (most recent call last):
File "tools/train_net.py", line 132, in <module>
main()
File "tools/train_net.py", line 114, in main
checkpoints = detectron.utils.train.train_model()
File "/home/lenovo/Detectron/detectron/detectron/utils/train.py", line 67, in train_model
workspace.RunNet(model.net.Proto().name)
File "/usr/local/lib/python2.7/dist-packages/caffe2/python/workspace.py", line 219, in RunNet
StringifyNetName(name), num_iter, allow_fail,
File "/usr/local/lib/python2.7/dist-packages/caffe2/python/workspace.py", line 180, in CallWithExceptionIntercept
return func(*args, **kwargs)
RuntimeError: [enforce fail at pybind_state.h:433] . Exception encountered running PythonOp function: ValueError: could not broadcast input array from shape (4) into shape (0)
At:
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(233): _expand_bbox_targets
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(174): _sample_rois
/home/lenovo/Detectron/detectron/detectron/roi_data/fast_rcnn.py(112): add_fast_rcnn_blobs
/home/lenovo/Detectron/detectron/detectron/ops/collect_and_distribute_fpn_rpn_proposals.py(62): forward
Error from operator:
input: "gpu_0/rpn_rois_fpn2" input: "gpu_0/rpn_rois_fpn3" input: "gpu_0/rpn_rois_fpn4" input: "gpu_0/rpn_rois_fpn5" input: "gpu_0/rpn_rois_fpn6" input: "gpu_0/rpn_roi_probs_fpn2" input: "gpu_0/rpn_roi_probs_fpn3" input: "gpu_0/rpn_roi_probs_fpn4" input: "gpu_0/rpn_roi_probs_fpn5" input: "gpu_0/rpn_roi_probs_fpn6" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois" output: "gpu_0/labels_int32" output: "gpu_0/bbox_targets" output: "gpu_0/bbox_inside_weights" output: "gpu_0/bbox_outside_weights" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_fpn2" output: "gpu_0/rois_fpn3" output: "gpu_0/rois_fpn4" output: "gpu_0/rois_fpn5" output: "gpu_0/rois_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "CollectAndDistributeFpnRpnProposalsOp:gpu_0/rpn_rois_fpn2,gpu_0/rpn_rois_fpn3,gpu_0/rpn_rois_fpn4,gpu_0/rpn_rois_fpn5,gpu_0/rpn_rois_fpn6,gpu_0/rpn_roi_probs_fpn2,gpu_0/rpn_roi_probs_fpn3,gpu_0/rpn_roi_probs_fpn4,gpu_0/rpn_roi_probs_fpn5,gpu_0/rpn_roi_probs_fpn6,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:5" } arg { name: "grad_output_indices" } device_option { device_type: 0 }Error from operator:
input: "gpu_0/rpn_rois_fpn2" input: "gpu_0/rpn_rois_fpn3" input: "gpu_0/rpn_rois_fpn4" input: "gpu_0/rpn_rois_fpn5" input: "gpu_0/rpn_rois_fpn6" input: "gpu_0/rpn_roi_probs_fpn2" input: "gpu_0/rpn_roi_probs_fpn3" input: "gpu_0/rpn_roi_probs_fpn4" input: "gpu_0/rpn_roi_probs_fpn5" input: "gpu_0/rpn_roi_probs_fpn6" input: "gpu_0/roidb" input: "gpu_0/im_info" output: "gpu_0/rois" output: "gpu_0/labels_int32" output: "gpu_0/bbox_targets" output: "gpu_0/bbox_inside_weights" output: "gpu_0/bbox_outside_weights" output: "gpu_0/mask_rois" output: "gpu_0/roi_has_mask_int32" output: "gpu_0/masks_int32" output: "gpu_0/rois_fpn2" output: "gpu_0/rois_fpn3" output: "gpu_0/rois_fpn4" output: "gpu_0/rois_fpn5" output: "gpu_0/rois_idx_restore_int32" output: "gpu_0/mask_rois_fpn2" output: "gpu_0/mask_rois_fpn3" output: "gpu_0/mask_rois_fpn4" output: "gpu_0/mask_rois_fpn5" output: "gpu_0/mask_rois_idx_restore_int32" name: "CollectAndDistributeFpnRpnProposalsOp:gpu_0/rpn_rois_fpn2,gpu_0/rpn_rois_fpn3,gpu_0/rpn_rois_fpn4,gpu_0/rpn_rois_fpn5,gpu_0/rpn_rois_fpn6,gpu_0/rpn_roi_probs_fpn2,gpu_0/rpn_roi_probs_fpn3,gpu_0/rpn_roi_probs_fpn4,gpu_0/rpn_roi_probs_fpn5,gpu_0/rpn_roi_probs_fpn6,gpu_0/roidb,gpu_0/im_info" type: "Python" arg { name: "grad_input_indices" } arg { name: "token" s: "forward:5" } arg { name: "grad_output_indices" } device_option { device_type: 1 device_id: 0 }frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*) + 0x76 (0x7f68b8086b46 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: <unknown function> + 0xae9c7 (0x7f68cb3599c7 in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #2: <unknown function> + 0xabbab (0x7f68cb356bab in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #3: <unknown function> + 0xeed4f (0x7f68cb399d4f in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #4: <unknown function> + 0xee27a (0x7f68cb39927a in /usr/local/lib/python2.7/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #5: caffe2::AsyncNetBase::run(int, int) + 0x154 (0x7f68ca0dc544 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #6: <unknown function> + 0x1311845 (0x7f68ca0e4845 in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #7: <unknown function> + 0x130c18b (0x7f68ca0df18b in /usr/local/lib/python2.7/dist-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #8: <unknown function> + 0xb8c80 (0x7f68f9404c80 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #9: <unknown function> + 0x76ba (0x7f68ffbbd6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #10: clone + 0x6d (0x7f68ff8f341d in /lib/x86_64-linux-gnu/libc.so.6)
System information
- Operating system: Ubuntu 16
- Compiler version: ?
- CUDA version: 10
- cuDNN version: 7
- NVIDIA driver version: 410
python --versionoutput: 2.7
Issue Analytics
- State:
- Created 5 years ago
- Comments:7
Top Results From Across the Web
ValueError: could not broadcast input array from shape ...
You're trying to create a new array out of a list of 3D arrays, so the final array could be 3 or 4D....
Read more >Python Error ValueError could not broadcast input array from ...
I have the following python code: big_array = np.zeros(shape=(100100), dtype=np.uint8) mini_square ... input array from shape (4) into shape ...
Read more >ValueError: could not broadcast input array from shape (2) into ...
I am trying to run stable_baseline alogs such as ppo1, ddpg and get this error: ValueError: could not broadcast input array from shape...
Read more >could not broadcast input array from shape (70) into shape (1)
I am working on a project that detects buildings on SpaceNet dataset by using Mask-RCNN. When I run this code: model.train(dataset_train, dataset_val, ...
Read more >Python Error ""ValueError: could not broadcast input array ...
Your data generator retrieves your labels as categorical and based on the error, I assume you have 4 classes.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
The question was from the ‘NUM_CLASS’ in the xxx.yaml file 😃
Maybe you are right. But I fixed it by another JSON file.