Training PartA2 network
See original GitHub issueHello @sshaoshuai,
Thank you for your great work!
I have trouble training PartA2 network.
Evaluating PartA2 and training PV-RCNN, PointRCNN, and SECOND work fine.
x = self.conv_input(input_sp_tensor) of UNetV2 gives nan values for some batches, and the nan value yields 0 max_overlap, 0 foreground and 0 background.
How can I resolve this issue?
Does anyone have a similar experience?
Here is the log.
Please note that my data set is Nusecenes exported to KITTI.
2020-09-07 00:43:50,426 INFO CUDA_VISIBLE_DEVICES=ALL
2020-09-07 00:43:50,426 INFO cfg_file /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models/PartA2.yaml
2020-09-07 00:43:50,426 INFO batch_size 1
2020-09-07 00:43:50,426 INFO epochs 85
2020-09-07 00:43:50,427 INFO workers 8
2020-09-07 00:43:50,427 INFO extra_tag default
2020-09-07 00:43:50,427 INFO ckpt /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/checkpoints/kitti/PartA2_7940.pth
2020-09-07 00:43:50,427 INFO pretrained_model None
2020-09-07 00:43:50,427 INFO launcher none
2020-09-07 00:43:50,427 INFO tcp_port 18888
2020-09-07 00:43:50,427 INFO sync_bn False
2020-09-07 00:43:50,427 INFO fix_random_seed False
2020-09-07 00:43:50,427 INFO ckpt_save_interval 1
2020-09-07 00:43:50,427 INFO local_rank 0
2020-09-07 00:43:50,427 INFO max_ckpt_save_num 30
2020-09-07 00:43:50,427 INFO merge_all_iters_to_one_epoch False
2020-09-07 00:43:50,427 INFO set_cfgs None
2020-09-07 00:43:50,427 INFO max_waiting_mins 0
2020-09-07 00:43:50,427 INFO start_epoch 0
2020-09-07 00:43:50,427 INFO save_to_file False
2020-09-07 00:43:50,427 INFO cfg.ROOT_DIR: /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet
2020-09-07 00:43:50,427 INFO cfg.LOCAL_RANK: 0
2020-09-07 00:43:50,427 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2020-09-07 00:43:50,427 INFO
cfg.DATA_CONFIG = edict()
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.DATASET: KittiDataset
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/kitti
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]
2020-09-07 00:43:50,427 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val
2020-09-07 00:43:50,427 INFO
cfg.DATA_CONFIG.INFO_PATH = edict()
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl']
2020-09-07 00:43:50,427 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl']
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2020-09-07 00:43:50,428 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:20', 'Pedestrian:15'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2020-09-07 00:43:50,428 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.05, 0.05, 0.1], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}]
2020-09-07 00:43:50,428 INFO cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml
2020-09-07 00:43:50,428 INFO
cfg.MODEL = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.NAME: PartA2Net
2020-09-07 00:43:50,428 INFO
cfg.MODEL.VFE = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.VFE.NAME: MeanVFE
2020-09-07 00:43:50,428 INFO
cfg.MODEL.BACKBONE_3D = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_3D.NAME: UNetV2
2020-09-07 00:43:50,428 INFO
cfg.MODEL.MAP_TO_BEV = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2020-09-07 00:43:50,428 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2020-09-07 00:43:50,428 INFO
cfg.MODEL.BACKBONE_2D = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2020-09-07 00:43:50,428 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2020-09-07 00:43:50,428 INFO
cfg.MODEL.DENSE_HEAD = edict()
2020-09-07 00:43:50,428 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2020-09-07 00:43:50,428 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2020-09-07 00:43:50,428 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2020-09-07 00:43:50,429 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2020-09-07 00:43:50,429 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,429 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2020-09-07 00:43:50,429 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2020-09-07 00:43:50,429 INFO
cfg.MODEL.POINT_HEAD = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.NAME: PointIntraPartOffsetHead
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.CLS_FC: []
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.PART_FC: []
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.CLASS_AGNOSTIC: True
2020-09-07 00:43:50,429 INFO
cfg.MODEL.POINT_HEAD.TARGET_CONFIG = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.TARGET_CONFIG.GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
2020-09-07 00:43:50,429 INFO
cfg.MODEL.POINT_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_REG: smooth-l1
2020-09-07 00:43:50,429 INFO
cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_cls_weight: 1.0
2020-09-07 00:43:50,429 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_part_weight: 1.0
2020-09-07 00:43:50,429 INFO
cfg.MODEL.ROI_HEAD = edict()
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NAME: PartA2FCHead
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.CLASS_AGNOSTIC: True
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.SHARED_FC: [256, 256, 256]
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.CLS_FC: [256, 256]
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.REG_FC: [256, 256]
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.DP_RATIO: 0.3
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.SEG_MASK_SCORE_THRESH: 0.3
2020-09-07 00:43:50,430 INFO
cfg.MODEL.ROI_HEAD.NMS_CONFIG = edict()
2020-09-07 00:43:50,430 INFO
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN = edict()
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_PRE_MAXSIZE: 9000
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_POST_MAXSIZE: 512
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_THRESH: 0.8
2020-09-07 00:43:50,430 INFO
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST = edict()
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_PRE_MAXSIZE: 1024
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_POST_MAXSIZE: 100
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_THRESH: 0.7
2020-09-07 00:43:50,430 INFO
cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL = edict()
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.POOL_SIZE: 12
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.NUM_FEATURES: 128
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.ROI_AWARE_POOL.MAX_POINTS_PER_VOXEL: 128
2020-09-07 00:43:50,430 INFO
cfg.MODEL.ROI_HEAD.TARGET_CONFIG = edict()
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.BOX_CODER: ResidualCoder
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.ROI_PER_IMAGE: 128
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.FG_RATIO: 0.5
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.SAMPLE_ROI_BY_EACH_CLASS: True
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_SCORE_TYPE: roi_iou
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_FG_THRESH: 0.75
2020-09-07 00:43:50,430 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH: 0.25
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH_LO: 0.1
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.HARD_BG_RATIO: 0.8
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.REG_FG_THRESH: 0.65
2020-09-07 00:43:50,431 INFO
cfg.MODEL.ROI_HEAD.LOSS_CONFIG = edict()
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CLS_LOSS: BinaryCrossEntropy
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.REG_LOSS: smooth-l1
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CORNER_LOSS_REGULARIZATION: True
2020-09-07 00:43:50,431 INFO
cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_cls_weight: 1.0
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_reg_weight: 1.0
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_corner_weight: 1.0
2020-09-07 00:43:50,431 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2020-09-07 00:43:50,431 INFO
cfg.MODEL.POST_PROCESSING = edict()
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2020-09-07 00:43:50,431 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.1
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2020-09-07 00:43:50,431 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2020-09-07 00:43:50,431 INFO
cfg.OPTIMIZATION = edict()
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.LR: 0.01
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9
2020-09-07 00:43:50,431 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.PCT_START: 0.4
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.LR_WARMUP: False
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2020-09-07 00:43:50,432 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2020-09-07 00:43:50,432 INFO cfg.TAG: PartA2
2020-09-07 00:43:50,432 INFO cfg.EXP_GROUP_PATH: home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models
2020-09-07 00:43:51,385 INFO Database filter by min points Car: 116656 => 47595
2020-09-07 00:43:51,391 INFO Database filter by min points Pedestrian: 52521 => 13192
2020-09-07 00:43:51,444 INFO Database filter by difficulty Car: 47595 => 33349
2020-09-07 00:43:51,458 INFO Database filter by difficulty Pedestrian: 13192 => 10978
2020-09-07 00:43:51,458 INFO Database filter by difficulty Cyclist: 0 => 0
2020-09-07 00:43:51,592 INFO Loading KITTI dataset
2020-09-07 00:43:52,198 INFO Total samples for KITTI dataset: 26291
2020-09-07 00:43:54,469 INFO ==> Loading parameters from checkpoint /home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/checkpoints/kitti/PartA2_7940.pth to CPU
2020-09-07 00:43:54,606 INFO ==> Done
2020-09-07 00:43:54,614 INFO PartA2Net(
(vfe): MeanVFE()
(backbone_3d): UNetV2(
(conv_input): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv1): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv2): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv3): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv4): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv_out): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv_up_t4): SparseBasicBlock(
(conv1): SubMConv3d()
(bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(relu): ReLU()
(conv2): SubMConv3d()
(bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
)
(conv_up_m4): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(inv_conv4): SparseSequential(
(0): SparseInverseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv_up_t3): SparseBasicBlock(
(conv1): SubMConv3d()
(bn1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(relu): ReLU()
(conv2): SubMConv3d()
(bn2): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
)
(conv_up_m3): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(inv_conv3): SparseSequential(
(0): SparseInverseConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv_up_t2): SparseBasicBlock(
(conv1): SubMConv3d()
(bn1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(relu): ReLU()
(conv2): SubMConv3d()
(bn2): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
)
(conv_up_m2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(inv_conv2): SparseSequential(
(0): SparseInverseConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv_up_t1): SparseBasicBlock(
(conv1): SubMConv3d()
(bn1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(relu): ReLU()
(conv2): SubMConv3d()
(bn2): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
)
(conv_up_m1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv5): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
)
(map_to_bev_module): HeightCompression()
(pfe): None
(backbone_2d): BaseBEVBackbone(
(blocks): ModuleList(
(0): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False)
(2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
(1): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False)
(2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
)
(deblocks): ModuleList(
(0): Sequential(
(0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): Sequential(
(0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
)
(dense_head): AnchorHeadSingle(
(cls_loss_func): SigmoidFocalClassificationLoss()
(reg_loss_func): WeightedSmoothL1Loss()
(dir_loss_func): WeightedCrossEntropyLoss()
(conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1))
(conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1))
(conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1))
)
(point_head): PointIntraPartOffsetHead(
(cls_loss_func): SigmoidFocalClassificationLoss()
(cls_layers): Sequential(
(0): Linear(in_features=16, out_features=1, bias=True)
)
(part_reg_layers): Sequential(
(0): Linear(in_features=16, out_features=3, bias=True)
)
)
(roi_head): PartA2FCHead(
(proposal_target_layer): ProposalTargetLayer()
(reg_loss_func): WeightedSmoothL1Loss()
(SA_modules): ModuleList()
(conv_part): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv_rpn): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(shared_fc_layer): Sequential(
(0): Conv1d(221184, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3, inplace=False)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
(7): Dropout(p=0.3, inplace=False)
(8): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(9): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(10): ReLU()
)
(cls_layers): Sequential(
(0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3, inplace=False)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv1d(256, 1, kernel_size=(1,), stride=(1,))
)
(reg_layers): Sequential(
(0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
(3): Dropout(p=0.3, inplace=False)
(4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False)
(5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv1d(256, 7, kernel_size=(1,), stride=(1,))
)
(roiaware_pool3d_layer): RoIAwarePool3d()
)
)
2020-09-07 00:43:54,615 INFO **********************Start training home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/cfgs/kitti_models/PartA2(default)**********************
epochs: 0%| | 0/5 [00:00<?, ?it/s]
train: 0%| | 0/26291 [00:00<?, ?it/s]
train: 0%| | 1/26291 [00:00<6:22:25, 1.15it/s]
epochs: 0%| | 0/5 [00:01<?, ?it/s, loss=4.68, lr=0.001]
train: 0%| | 2/26291 [00:01<4:59:45, 1.46it/s, total_it=12401]
epochs: 0%| | 0/5 [00:01<?, ?it/s, loss=3.62, lr=0.001]
train: 0%| | 3/26291 [00:01<4:02:46, 1.80it/s, total_it=12402]
epochs: 0%| | 0/5 [00:01<?, ?it/s, loss=4.07, lr=0.001]
train: 0%| | 4/26291 [00:01<3:22:12, 2.17it/s, total_it=12403]
epochs: 0%| | 0/5 [00:01<?, ?it/s, loss=3.53, lr=0.001]
train: 0%| | 5/26291 [00:01<2:54:01, 2.52it/s, total_it=12404]
epochs: 0%| | 0/5 [00:02<?, ?it/s, loss=4.19, lr=0.001]
train: 0%| | 6/26291 [00:02<2:31:47, 2.89it/s, total_it=12405]
epochs: 0%| | 0/5 [00:02<?, ?it/s, loss=2.52, lr=0.001]
train: 0%| | 7/26291 [00:02<2:16:11, 3.22it/s, total_it=12406]
epochs: 0%| | 0/5 [00:02<?, ?it/s, loss=2.7, lr=0.001]
train: 0%| | 8/26291 [00:02<2:05:12, 3.50it/s, total_it=12407]
epochs: 0%| | 0/5 [00:02<?, ?it/s, loss=3.27, lr=0.001]
train: 0%| | 9/26291 [00:02<1:56:13, 3.77it/s, total_it=12408]
epochs: 0%| | 0/5 [00:03<?, ?it/s, loss=4.17, lr=0.001]
train: 0%| | 10/26291 [00:02<1:50:29, 3.96it/s, total_it=12409]
epochs: 0%| | 0/5 [00:03<?, ?it/s, loss=3.17, lr=0.001]
train: 0%| | 11/26291 [00:03<1:47:31, 4.07it/s, total_it=12410]
epochs: 0%| | 0/5 [00:03<?, ?it/s, loss=2.57, lr=0.001]
train: 0%| | 12/26291 [00:03<1:44:55, 4.17it/s, total_it=12411]
epochs: 0%| | 0/5 [00:03<?, ?it/s, loss=4.33, lr=0.001]
train: 0%| | 13/26291 [00:03<1:43:06, 4.25it/s, total_it=12412]
epochs: 0%| | 0/5 [00:04<?, ?it/s, loss=2.88, lr=0.001]
train: 0%| | 14/26291 [00:03<1:41:09, 4.33it/s, total_it=12413]
epochs: 0%| | 0/5 [00:04<?, ?it/s, loss=4.2, lr=0.001]
train: 0%| | 15/26291 [00:04<1:40:32, 4.36it/s, total_it=12414]
epochs: 0%| | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Warning: Sparse_Idx_Shape(torch.Size([0, 4]))
train: 0%| | 16/26291 [00:04<1:35:34, 4.58it/s, total_it=12415]
epochs: 0%| | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
maxoverlaps:(min=nan, max=nan)
ERROR: FG=0, BG=0
epochs: 0%| | 0/5 [00:04<?, ?it/s, loss=nan, lr=0.001]
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/home/hcyoo/Programs/pycharm/pycharm-professional-2020.2/pycharm-2020.2/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/home/hcyoo/Programs/pycharm/pycharm-professional-2020.2/pycharm-2020.2/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train.py", line 206, in <module>
main()
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train.py", line 178, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train_utils/train_utils.py", line 93, in train_model
dataloader_iter=dataloader_iter
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/tools/train_utils/train_utils.py", line 38, in train_one_epoch
loss, tb_dict, disp_dict = model_func(model, batch)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/__init__.py", line 30, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File "/home/hcyoo/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/detectors/PartA2_net.py", line 11, in forward
batch_dict = cur_module(batch_dict)
File "/home/hcyoo/anaconda3/envs/pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/partA2_head.py", line 175, in forward
targets_dict = self.assign_targets(batch_dict)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/roi_head_template.py", line 104, in assign_targets targets_dict = self.proposal_target_layer.forward(batch_dict)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 33, in forward
batch_dict=batch_dict
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 107, in sample_rois_for_rcnn
sampled_inds = self.subsample_rois(max_overlaps=max_overlaps)
File "/home/hcyoo/anaconda3/envs/pcdet/OpenPCDet/pcdet/models/roi_heads/target_assigner/proposal_target_layer.py", line 159, in subsample_rois
raise NotImplementedError
NotImplementedError
Issue Analytics
- State:
- Created 3 years ago
- Comments:7
Top Results From Across the Web
Making the Pass, Part 2: Training a Neural Network with KNIME
Making the Pass, Part 2: Training a Neural Network with KNIME. A fully codeless approach using KNIME's Keras integration. As first published on...
Read more >Neural networks made easy (Part 2): Network training ... - MQL5
In this second article, we will continue to study neural networks and will consider an example of using our created CNet class in...
Read more >CNNs, Part 2: Training a Convolutional Neural Network
CNNs, Part 2: Training a Convolutional Neural Network ... introductions to Convolutional Neural Networks (CNNs) lack: how to train a CNN, ...
Read more >Improving Quality in Virtual Teaching Across a Network: Part 2
Jeanine Zitta knew teachers faced a steep learning curve. In the spring, she'd seen how teachers in her network of nine elementary schools...
Read more >Crash Course: Neural Networks Part 2 — Intuition
This has been it for Part 2 of the Neural Network Crash Course. I hope you now have an intuitive understanding of how...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

thanks for your reply! Appreciate!
hi, i used your way to solve the problem,but failed. do you have some other methods?