SSD_MobileNetv1_COCO label_filename incorrect classification

Description When running ssd_mobilenetv1_coco in triton, and specifying the class labels in the model config, it seems that the labels are not assigned correctly.

Triton Information What version of Triton are you using? NVIDIA Release 20.03 (build 11042949)

Are you using the Triton container or did you build it yourself? Using triton container

To Reproduce Steps to reproduce the behavior. My model config for ssd_mobilenetv1_coco is as follows:

name: "ssd_mobilenet_coco"
platform: "tensorflow_graphdef"
max_batch_size: 1
input [
  {
    name: "image_tensor"
    data_type: TYPE_UINT8
    format: FORMAT_NHWC
    dims: [ 300, 300, 3 ]
  }
]
output [
  {
    name: "detection_boxes"
    data_type: TYPE_FP32
    dims: [ 100, 4 ]
  },
  {
    name: "detection_scores"
    data_type: TYPE_FP32
    dims: [ 100 ]
  },
  {
    name: "num_detections"
    data_type: TYPE_FP32
    dims: [ 1 ]
    reshape: { shape: [ ] }
  },
  {
    name: "detection_classes"
    data_type: TYPE_FP32
    dims: [ 100 ]
    label_filename: "ssd_mobilenet_coco.classes"
  }
]

A snippet of my labels file ssd_mobilenet_coco.classes:

0 unlabeled
1 person
2 bicycle
3 car

With a properly formatted image I make the request:

result = ctx.run(
          { input_name : (preprocess(img, format, dtype, c, h, w, args.scaling),) },
          { output_names[3] : (InferContext.ResultFormat.CLASS, args.classes),
            output_names[2] : InferContext.ResultFormat.RAW,
            output_names[1] : InferContext.ResultFormat.RAW,
            output_names[0] : InferContext.ResultFormat.RAW })

Which yields the incorrect labels as follows:

{'detection_classes': [[(61, 67.0, '61 cake'), (18, 42.0, '18 dog'), (73, 42.0, '73 laptop'), (49, 42.0, '49 knife'), (36, 41.0, '36 snowboard'), (35, 41.0, '35 skis'), (94, 41.0, '94 branch'), (83, 41.0, '83 blender'), (87, 41.0, '87 scissors'), (24, 41.0, '24 zebra')]], 'num_detections': [array([100.], dtype=float32)]

If I rather make a request like the following, using the RAW format rather than CLASS:

result = ctx.run(
          { input_name : (preprocess(img, format, dtype, c, h, w, args.scaling),) },
          { output_names[3] : InferContext.ResultFormat.RAW,
            output_names[2] : InferContext.ResultFormat.RAW,
            output_names[1] : InferContext.ResultFormat.RAW,
            output_names[0] : InferContext.ResultFormat.RAW })

I get the following output:

'detection_classes': [array([ 1.,  1.,  3.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  3.,  1.,  1.,
        1.,  1.,  3.,  1.,  1., 42., 33.,  1., 10.,  1.,  1., 41.,  1.,
       31.,  1.,  1., 31., 10.,  1.,  1., 31.,  1., 41., 41.,  8., 10.,
        1.,  1.,  3.,  1., 10., 10.,  8., 10., 41., 31., 42.,  1.,  1.,
       31.,  1.,  1.,  1., 10.,  1.,  1.,  3.,  1., 67.,  1., 31.,  3.,
        1.,  1.,  1.,  1.,  1.,  1.,  3., 10., 42., 31., 31.,  1.,  1.,
        8.,  1.,  1., 31.,  1., 41., 31., 10., 31., 41., 33., 31., 31.,
        1.,  1.,  1., 41.,  1., 31.,  1.,  1.,  1.], dtype=float32)],

Which clearly should map to mostly people and cars.

Expected behavior I would expect the indexes in the detection classes tensor above to be mapped to my classes file appropriately. I’m not sure how the list with cake, dog, etc is being generated.

[Rusteam]

Thanks @Rusteam how did you do it exactly, please?

In my case I have TF object detection model, and Triton is treating its output class Tensor as a probability values and map their indexes to the the label_map, which is wrong, because the output tensor shape (100 in my case) is independent of the number of classes, and those values are already indexes w.r.t their label_map. Triton should take the values as they are and map them.

I have an ensemble model that connects preprocessing, yolov5 model and postprocessing. Additionally, I sent class probs from post processing to an identity matrix that has a labels file.

[mhbassel]

Thanks @Rusteam how did you do it exactly, please?

In my case I have TF object detection model, and Triton is treating its output class Tensor as a probability values and map their indexes to the the label_map, which is wrong, because the output tensor shape (100 in my case) is independent of the number of classes, and those values are already indexes w.r.t their label_map. Triton should take the values as they are and map them.