Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Evaluate on pandaset get very low precision

See original GitHub issue

@sshaoshuai @lindahua @jihanyang Thank you so much for your awesome work.

I want to use the model trained on Kitti to evaluate the accuracy on pandaset.

in pandaset_dataset.py, I changed evaluation function just like kitti evaluation.

I searched for a lot of methods on the Internet, mainly there are two methods.

  1. Import Pandaset by dataloader, and modify the evaluation function.
  2. Convert Pandaset to kitti format for evaluation.

I tried to use dataloader and imitate kitti’s evaluation function. I am getting very low precision, can you tell me why?Maybe how to get the right answer,please.


        # self.logger.warning('Evaluation is not implemented for Pandaset as there is no official one. ' +
        #                     'Returning an empty evaluation result.')
        if 'annos' not in self.pandaset_infos[0].keys():
            return None, {}

        from .kitti_object_eval_python import eval as kitti_eval

        eval_det_annos = copy.deepcopy(det_annos)
        eval_gt_annos = [copy.deepcopy(info['annos']) for info in self.pandaset_infos]
        ap_result_str, ap_dict = kitti_eval.get_official_eval_result(eval_gt_annos, eval_det_annos, class_names)

        return ap_result_str, ap_dict

And I also changed pv_rcnn.yaml

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG:
    _BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['placeholder']
        AUG_CONFIG_LIST:
            - NAME: gt_sampling
              USE_ROAD_PLANE: True
              DB_INFO_PATH:
                  - kitti_dbinfos_train.pkl
              PREPARE: {
                 filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
                 filter_by_difficulty: [-1],
              }

              SAMPLE_GROUPS: ['Car:15','Pedestrian:10', 'Cyclist:10']
              NUM_POINT_FEATURES: 4
              DATABASE_WITH_FAKELIDAR: False
              REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
              LIMIT_WHOLE_SCENE: False

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x']

            - NAME: random_world_rotation
              WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL:
    NAME: PVRCNN

    VFE:
        NAME: MeanVFE

    BACKBONE_3D:
        NAME: VoxelBackBone8x

    MAP_TO_BEV:
        NAME: HeightCompression
        NUM_BEV_FEATURES: 256

    BACKBONE_2D:
        NAME: BaseBEVBackbone

        LAYER_NUMS: [5, 5]
        LAYER_STRIDES: [1, 2]
        NUM_FILTERS: [128, 256]
        UPSAMPLE_STRIDES: [1, 2]
        NUM_UPSAMPLE_FILTERS: [256, 256]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'Car',
                'anchor_sizes': [[3.9, 1.6, 1.56]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-1.78],
                'align_center': False,
                'feature_map_stride': 8,
                'matched_threshold': 0.6,
                'unmatched_threshold': 0.45
            },
            {
                'class_name': 'Pedestrian',
                'anchor_sizes': [[0.8, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 8,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            },
            {
                'class_name': 'Cyclist',
                'anchor_sizes': [[1.76, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 8,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    PFE:
        NAME: VoxelSetAbstraction
        POINT_SOURCE: raw_points
        NUM_KEYPOINTS: 2048
        NUM_OUTPUT_FEATURES: 128
        SAMPLE_METHOD: FPS

        FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points']
        SA_LAYER:
            raw_points:
                MLPS: [[16, 16], [16, 16]]
                POOL_RADIUS: [0.4, 0.8]
                NSAMPLE: [16, 16]
            x_conv1:
                DOWNSAMPLE_FACTOR: 1
                MLPS: [[16, 16], [16, 16]]
                POOL_RADIUS: [0.4, 0.8]
                NSAMPLE: [16, 16]
            x_conv2:
                DOWNSAMPLE_FACTOR: 2
                MLPS: [[32, 32], [32, 32]]
                POOL_RADIUS: [0.8, 1.2]
                NSAMPLE: [16, 32]
            x_conv3:
                DOWNSAMPLE_FACTOR: 4
                MLPS: [[64, 64], [64, 64]]
                POOL_RADIUS: [1.2, 2.4]
                NSAMPLE: [16, 32]
            x_conv4:
                DOWNSAMPLE_FACTOR: 8
                MLPS: [[64, 64], [64, 64]]
                POOL_RADIUS: [2.4, 4.8]
                NSAMPLE: [16, 32]

    POINT_HEAD:
        NAME: PointHeadSimple
        CLS_FC: [256, 256]
        CLASS_AGNOSTIC: True
        USE_POINT_FEATURES_BEFORE_FUSION: True
        TARGET_CONFIG:
            GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
        LOSS_CONFIG:
            LOSS_REG: smooth-l1
            LOSS_WEIGHTS: {
                'point_cls_weight': 1.0,
            }

    ROI_HEAD:
        NAME: PVRCNNHead
        CLASS_AGNOSTIC: True

        SHARED_FC: [256, 256]
        CLS_FC: [256, 256]
        REG_FC: [256, 256]
        DP_RATIO: 0.3

        NMS_CONFIG:
            TRAIN:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 9000
                NMS_POST_MAXSIZE: 512
                NMS_THRESH: 0.8
            TEST:
                NMS_TYPE: nms_gpu
                MULTI_CLASSES_NMS: False
                NMS_PRE_MAXSIZE: 1024
                NMS_POST_MAXSIZE: 100
                NMS_THRESH: 0.7

        ROI_GRID_POOL:
            GRID_SIZE: 6
            MLPS: [[64, 64], [64, 64]]
            POOL_RADIUS: [0.8, 1.6]
            NSAMPLE: [16, 16]
            POOL_METHOD: max_pool

        TARGET_CONFIG:
            BOX_CODER: ResidualCoder
            ROI_PER_IMAGE: 128
            FG_RATIO: 0.5

            SAMPLE_ROI_BY_EACH_CLASS: True
            CLS_SCORE_TYPE: roi_iou

            CLS_FG_THRESH: 0.75
            CLS_BG_THRESH: 0.25
            CLS_BG_THRESH_LO: 0.1
            HARD_BG_RATIO: 0.8

            REG_FG_THRESH: 0.55

        LOSS_CONFIG:
            CLS_LOSS: BinaryCrossEntropy
            REG_LOSS: smooth-l1
            CORNER_LOSS_REGULARIZATION: True
            LOSS_WEIGHTS: {
                'rcnn_cls_weight': 1.0,
                'rcnn_reg_weight': 1.0,
                'rcnn_corner_weight': 1.0,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.1
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    BATCH_SIZE_PER_GPU: 2
    NUM_EPOCHS: 80

    OPTIMIZER: adam_onecycle
    LR: 0.01
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10

Finally I use this command python test.py --cfg_file cfgs/pandaset_models/pv_rcnn.yaml --batch_size 1 --ckpt pv_rcnn_8369.pth \--save_to_file

Got

python test.py --cfg_file cfgs/pandaset_models/pv_rcnn.yaml --batch_size 1 --ckpt pv_rcnn_8369.pth     --save_to_file
/home/work/anaconda3/envs/openpcdet-new/lib/python3.7/site-packages/skimage/io/manage_plugins.py:23: UserWarning: Your installed pillow version is < 8.1.2. Several security issues (CVE-2021-27921, CVE-2021-25290, CVE-2021-25291, CVE-2021-25293, and more) have been fixed in pillow 8.1.2 or higher. We recommend to upgrade this library.
  from .collection import imread_collection_wrapper
2022-02-16 07:14:36,886   INFO  **********************Start logging**********************
2022-02-16 07:14:36,886   INFO  CUDA_VISIBLE_DEVICES=ALL
2022-02-16 07:14:36,886   INFO  cfg_file         cfgs/pandaset_models/pv_rcnn.yaml
2022-02-16 07:14:36,886   INFO  batch_size       1
2022-02-16 07:14:36,886   INFO  workers          4
2022-02-16 07:14:36,886   INFO  extra_tag        default
2022-02-16 07:14:36,886   INFO  ckpt             pv_rcnn_8369.pth
2022-02-16 07:14:36,886   INFO  launcher         none
2022-02-16 07:14:36,886   INFO  tcp_port         18888
2022-02-16 07:14:36,886   INFO  local_rank       0
2022-02-16 07:14:36,886   INFO  set_cfgs         None
2022-02-16 07:14:36,886   INFO  max_waiting_mins 30
2022-02-16 07:14:36,886   INFO  start_epoch      0
2022-02-16 07:14:36,886   INFO  eval_tag         default
2022-02-16 07:14:36,886   INFO  eval_all         False
2022-02-16 07:14:36,886   INFO  ckpt_dir         None
2022-02-16 07:14:36,886   INFO  save_to_file     True
2022-02-16 07:14:36,886   INFO  cfg.ROOT_DIR: /home/work/work/OpenPCDet
2022-02-16 07:14:36,887   INFO  cfg.LOCAL_RANK: 0
2022-02-16 07:14:36,887   INFO  cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2022-02-16 07:14:36,887   INFO  
cfg.DATA_CONFIG = edict()
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.DATASET: PandasetDataset
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.DATA_PATH: ../data/pandaset
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-70, -40, -3, 70, 40, 1]
2022-02-16 07:14:36,887   INFO  
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.DATA_SPLIT.train: train
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.DATA_SPLIT.test: val
2022-02-16 07:14:36,887   INFO  
cfg.DATA_CONFIG.SEQUENCES = edict()
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.SEQUENCES.train: ['014', '050', '079', '048', '093', '091', '063', '104', '100', '092', '012', '047', '018', '006', '099', '085', '035', '041', '052', '105', '030', '113', '002', '084', '028', '119', '044', '005', '102', '034', '077', '064', '067', '058', '019', '015', '037', '095', '120', '066', '023', '071', '117', '098', '139', '038', '116', '046', '088', '089', '040', '033', '016', '024', '122', '039', '158', '069', '124', '123', '106']
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.SEQUENCES.val: ['045', '059', '055', '051', '020', '097', '073', '043', '003', '101', '027', '056', '011', '078', '080', '109', '042', '021', '094', '057']
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.SEQUENCES.test: ['074', '004', '086', '062', '068', '008', '001', '110', '053', '115', '054', '065', '017', '103', '072', '013', '029', '090', '112', '149', '070', '032']
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.LIDAR_DEVICE: 0
2022-02-16 07:14:36,887   INFO  
cfg.DATA_CONFIG.INFO_PATH = edict()
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.INFO_PATH.train: ['pandaset_infos_train.pkl']
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.INFO_PATH.test: ['pandaset_infos_val.pkl']
2022-02-16 07:14:36,887   INFO  
cfg.DATA_CONFIG.TRAINING_CATEGORIES = edict()
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Car: Car
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Pickup Truck: Car
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Medium-sized Truck: Truck
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Semi-truck: Truck
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Towed Object: Other Vehicle
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Motorcycle: Motorcycle
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Other Vehicle - Construction Vehicle: Other Vehicle
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Other Vehicle - Uncommon: Other Vehicle
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Other Vehicle - Pedicab: Other Vehicle
2022-02-16 07:14:36,887   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Emergency Vehicle: Other Vehicle
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Bus: Bus
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Bicycle: Bicycle
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Pedestrian: Pedestrian
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Pedestrian with Object: Pedestrian
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.TRAINING_CATEGORIES.Animals - Other: Animal
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.FOV_POINTS_ONLY: False
2022-02-16 07:14:36,888   INFO  
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': True, 'DB_INFO_PATH': ['pandaset_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': False}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2022-02-16 07:14:36,888   INFO  
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.05, 0.05, 0.1], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}]
2022-02-16 07:14:36,888   INFO  cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/pandaset_dataset.yaml
2022-02-16 07:14:36,888   INFO  
cfg.MODEL = edict()
2022-02-16 07:14:36,888   INFO  cfg.MODEL.NAME: PVRCNN
2022-02-16 07:14:36,888   INFO  
cfg.MODEL.VFE = edict()
2022-02-16 07:14:36,888   INFO  cfg.MODEL.VFE.NAME: MeanVFE
2022-02-16 07:14:36,888   INFO  
cfg.MODEL.BACKBONE_3D = edict()
2022-02-16 07:14:36,888   INFO  cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x
2022-02-16 07:14:36,888   INFO  
cfg.MODEL.MAP_TO_BEV = edict()
2022-02-16 07:14:36,888   INFO  cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2022-02-16 07:14:36,888   INFO  cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2022-02-16 07:14:36,888   INFO  
cfg.MODEL.BACKBONE_2D = edict()
2022-02-16 07:14:36,888   INFO  cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2022-02-16 07:14:36,888   INFO  cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2022-02-16 07:14:36,889   INFO  cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2022-02-16 07:14:36,889   INFO  cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2022-02-16 07:14:36,889   INFO  cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2022-02-16 07:14:36,889   INFO  cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2022-02-16 07:14:36,889   INFO  
cfg.MODEL.DENSE_HEAD = edict()
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2022-02-16 07:14:36,889   INFO  
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2022-02-16 07:14:36,889   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2022-02-16 07:14:36,889   INFO  
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2022-02-16 07:14:36,889   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2022-02-16 07:14:36,889   INFO  
cfg.MODEL.PFE = edict()
2022-02-16 07:14:36,889   INFO  cfg.MODEL.PFE.NAME: VoxelSetAbstraction
2022-02-16 07:14:36,889   INFO  cfg.MODEL.PFE.POINT_SOURCE: raw_points
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.NUM_KEYPOINTS: 2048
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.NUM_OUTPUT_FEATURES: 128
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SAMPLE_METHOD: FPS
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points']
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER = edict()
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER.raw_points = edict()
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.raw_points.MLPS: [[16, 16], [16, 16]]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.raw_points.POOL_RADIUS: [0.4, 0.8]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.raw_points.NSAMPLE: [16, 16]
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER.x_conv1 = edict()
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv1.DOWNSAMPLE_FACTOR: 1
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv1.MLPS: [[16, 16], [16, 16]]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv1.POOL_RADIUS: [0.4, 0.8]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv1.NSAMPLE: [16, 16]
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER.x_conv2 = edict()
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv2.DOWNSAMPLE_FACTOR: 2
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv2.MLPS: [[32, 32], [32, 32]]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv2.POOL_RADIUS: [0.8, 1.2]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv2.NSAMPLE: [16, 32]
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER.x_conv3 = edict()
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv3.DOWNSAMPLE_FACTOR: 4
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv3.MLPS: [[64, 64], [64, 64]]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv3.POOL_RADIUS: [1.2, 2.4]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv3.NSAMPLE: [16, 32]
2022-02-16 07:14:36,890   INFO  
cfg.MODEL.PFE.SA_LAYER.x_conv4 = edict()
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv4.DOWNSAMPLE_FACTOR: 8
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv4.MLPS: [[64, 64], [64, 64]]
2022-02-16 07:14:36,890   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv4.POOL_RADIUS: [2.4, 4.8]
2022-02-16 07:14:36,891   INFO  cfg.MODEL.PFE.SA_LAYER.x_conv4.NSAMPLE: [16, 32]
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.POINT_HEAD = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.NAME: PointHeadSimple
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.CLS_FC: [256, 256]
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.CLASS_AGNOSTIC: True
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.USE_POINT_FEATURES_BEFORE_FUSION: True
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.POINT_HEAD.TARGET_CONFIG = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.TARGET_CONFIG.GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.POINT_HEAD.LOSS_CONFIG = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_REG: smooth-l1
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_cls_weight: 1.0
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.ROI_HEAD = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NAME: PVRCNNHead
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.CLASS_AGNOSTIC: True
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.SHARED_FC: [256, 256]
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.CLS_FC: [256, 256]
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.REG_FC: [256, 256]
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.DP_RATIO: 0.3
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG = edict()
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_TYPE: nms_gpu
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.MULTI_CLASSES_NMS: False
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_PRE_MAXSIZE: 9000
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_POST_MAXSIZE: 512
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_THRESH: 0.8
2022-02-16 07:14:36,891   INFO  
cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST = edict()
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_TYPE: nms_gpu
2022-02-16 07:14:36,891   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.MULTI_CLASSES_NMS: False
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_PRE_MAXSIZE: 1024
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_POST_MAXSIZE: 100
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_THRESH: 0.7
2022-02-16 07:14:36,892   INFO  
cfg.MODEL.ROI_HEAD.ROI_GRID_POOL = edict()
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.GRID_SIZE: 6
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.MLPS: [[64, 64], [64, 64]]
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_RADIUS: [0.8, 1.6]
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.NSAMPLE: [16, 16]
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_METHOD: max_pool
2022-02-16 07:14:36,892   INFO  
cfg.MODEL.ROI_HEAD.TARGET_CONFIG = edict()
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.BOX_CODER: ResidualCoder
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.ROI_PER_IMAGE: 128
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.FG_RATIO: 0.5
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.SAMPLE_ROI_BY_EACH_CLASS: True
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_SCORE_TYPE: roi_iou
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_FG_THRESH: 0.75
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH: 0.25
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH_LO: 0.1
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.HARD_BG_RATIO: 0.8
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.TARGET_CONFIG.REG_FG_THRESH: 0.55
2022-02-16 07:14:36,892   INFO  
cfg.MODEL.ROI_HEAD.LOSS_CONFIG = edict()
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CLS_LOSS: BinaryCrossEntropy
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.REG_LOSS: smooth-l1
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CORNER_LOSS_REGULARIZATION: True
2022-02-16 07:14:36,892   INFO  
cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_cls_weight: 1.0
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_reg_weight: 1.0
2022-02-16 07:14:36,892   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_corner_weight: 1.0
2022-02-16 07:14:36,893   INFO  cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2022-02-16 07:14:36,893   INFO  
cfg.MODEL.POST_PROCESSING = edict()
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2022-02-16 07:14:36,893   INFO  
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.1
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2022-02-16 07:14:36,893   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2022-02-16 07:14:36,893   INFO  
cfg.OPTIMIZATION = edict()
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 2
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.NUM_EPOCHS: 80
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.LR: 0.01
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.MOMENTUM: 0.9
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.PCT_START: 0.4
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.DIV_FACTOR: 10
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.LR_DECAY: 0.1
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.LR_CLIP: 1e-07
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.LR_WARMUP: False
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2022-02-16 07:14:36,893   INFO  cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2022-02-16 07:14:36,893   INFO  cfg.TAG: pv_rcnn
2022-02-16 07:14:36,894   INFO  cfg.EXP_GROUP_PATH: pandaset_models
2022-02-16 07:14:37,184   INFO  Loading PandaSet dataset
2022-02-16 07:14:37,185   INFO  Total samples for PandaSet dataset (test): 1600
2022-02-16 07:14:38,637   INFO  ==> Loading parameters from checkpoint pv_rcnn_8369.pth to GPU
2022-02-16 07:14:38,679   INFO  ==> Done (loaded 367/367)
2022-02-16 07:14:38,700   INFO  *************** EPOCH 8369 EVALUATION *****************
eval:   0%|                                                                                    | 0/1600 [00:00<?, ?it/s]../pcdet/models/roi_heads/pvrcnn_head.py:126: UserWarning: This overload of nonzero is deprecated:
nonzero()
Consider using one of the following signatures instead:
nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729047590/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  dense_idx = faked_features.nonzero()  # (N, 3) [x_idx, y_idx, z_idx]
eval: 100%|█████████████████████████████████████| 1600/1600 [08:09<00:00,  3.27it/s, recall_0.3=(28743, 29004) / 229468]
2022-02-16 07:22:47,887   INFO  *************** Performance of EPOCH 8369 *****************
2022-02-16 07:22:47,887   INFO  Generate label finished(sec_per_example: 0.3057 second).
2022-02-16 07:22:47,887   INFO  recall_roi_0.3: 0.125259
2022-02-16 07:22:47,887   INFO  recall_rcnn_0.3: 0.126397
2022-02-16 07:22:47,888   INFO  recall_roi_0.5: 0.100062
2022-02-16 07:22:47,888   INFO  recall_rcnn_0.5: 0.106080
2022-02-16 07:22:47,888   INFO  recall_roi_0.7: 0.030183
2022-02-16 07:22:47,888   INFO  recall_rcnn_0.7: 0.040332
2022-02-16 07:22:47,889   INFO  Average predicted number of objects(1600 samples): 14.273
2022-02-16 07:22:48,275   INFO  None
2022-02-16 07:22:48,275   INFO  Result is save to /home/work/work/OpenPCDet/output/pandaset_models/pv_rcnn/default/eval/epoch_8369/val/default
2022-02-16 07:22:48,276   INFO  ****************Evaluation done.*****************

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:10

github_iconTop GitHub Comments

2reactions
LouisSFcommented, Feb 18, 2022

Hi, I think that this may be due to the fact that you use a model trained on Kitti, which uses different reference coordinates than Pandaset,in Kitti the x axis points in front of the car, whereas in Pandaset it points to the left. You may also have to adjust the height of the lidar point clouds in pandaset to account for the difference in Lidar sensors, though I haven’t checked this precisely.

1reaction
Leozyc-wasedacommented, Feb 18, 2022

@LouisSF Thank you for your reply.

Are you referring to the second method I used to convert Pandaset data into kitti ? This reminds me, I will try to adjust the coordinates and use demo.py to see the result.

also, i noticed kitti point cloud range is [0, -40, -3, 70.4, 40, 1] ,

https://github.com/open-mmlab/OpenPCDet/blob/6a1495ca610186d3b9d5142a4a6506b8dc78bce1/tools/cfgs/dataset_configs/kitti_dataset.yaml#L4

for pandaset is [-70, -40, -3, 70, 40, 1] , https://github.com/open-mmlab/OpenPCDet/blob/6a1495ca610186d3b9d5142a4a6506b8dc78bce1/tools/cfgs/dataset_configs/pandaset_dataset.yaml#L4

I’ve been trying to test pandaset with dataloader. In the case of only detecting the car, good results are achieved, but I still don’t know why cyclists and pedestrian are not detected.

I tried to change Xmin from 0 to -70, the detection result of pv-rcnn did not change, it seems that the pv-rcnn model trained on kitti only detected the part from 0 to 70, and the part from -70 to 0 was not detected at all.

If all objects from -70 to 0 cannot be detected, is it due to the use of the pv-rcnn model trained on Kitti?

https://arxiv.org/pdf/2112.12610.pdf I read pandaset’s paper and found that they were also tested with pv-rcnn on openpcdet, I tried to imitate their configuration, and the detection results were very bad. I am retraining pv-rcnn with pandaset and it looks like it will take some time.

Read more comments on GitHub >

github_iconTop Results From Across the Web

High Recall - Low Precision for unbalanced dataset
When predicting I get a low precision (0.47) for the minority class in the validation set; recall is 0.88. I tried to use...
Read more >
Advanced Sensor Suite Dataset for Autonomous Driving - arXiv
In this paper, we introduce PandaSet, the first dataset produced by a complete, high-precision autonomous vehicle sensor kit.
Read more >
The PandaSet Dataset – Weights & Biases - WandB
Models can be evaluated using Average Precision. LiDAR Point Cloud Segmentation. The PandaSet Dataset also provides the ground truth labels for ...
Read more >
Getting good accuracy but very low precision for classification ...
I think I understand what is going on. Consider a dummy classifier that would return the majority class for each sample of your...
Read more >
PandaSet: Advanced Sensor Suite Dataset for Autonomous ...
LiDAR has become a standard sensor for autonomous driving applications a ... named PandaSet, which provides a complete kit of high-precision ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Training PV-RCNN++ , CUDA kernel failed : invalid configuration argument

Next page

Use pip instead of the deprecated easy_install