struct.error: unpack_from requires a buffer of at least 274435 bytes

I’m getting this error while decoding a TYPE_STRING output response from the Triton Server. Config.pbtxt looks like this:

name: "vehicleRec_postprocess"
backend: "python"
max_batch_size: 0
input [
    {
        name: "model_output"
        data_type: TYPE_FP32
        dims: [-1, -1]
    },
    {
        name: "resized_image_shape"
        data_type: TYPE_INT64
        dims: [2]
    },
    {
        name: "original_image_shape"
        data_type: TYPE_INT64
        dims: [2]
    }
]
output [
    {
        name: "final_output"
        data_type: TYPE_STRING
        dims: [-1,-1,-1]
    }
]

It’s actually is part of an ensemble, but this is the last model of that ensemble. The sample output I’m getting is:

[array([[[b'925', b'369'],
        [b'1013', b'395'],
        [b'bus', b'bus']],

       [[b'515', b'542'],
        [b'571', b'592'],
        [b'rickshaw', b'rickshaw']],

       [[b'458', b'357'],
        [b'508', b'397'],
        [b'bus', b'bus']],

       [[b'868', b'388'],
        [b'900', b'421'],
        [b'car', b'car']],

       [[b'1021', b'412'],
        [b'1113', b'461'],
        [b'bus', b'bus']],

       [[b'593', b'364'],
        [b'639', b'416'],
        [b'van', b'van']],

       [[b'913', b'324'],
        [b'982', b'364'],
        [b'bus', b'bus']],

       [[b'794', b'411'],
        [b'833', b'445'],
        [b'car', b'car']],

       [[b'282', b'460'],
        [b'552', b'641'],
        [b'bus', b'bus']],

       [[b'934', b'403'],
        [b'1104', b'470'],
        [b'bus', b'bus']],

       [[b'813', b'589'],
        [b'851', b'663'],
        [b'motorbike', b'motorbike']],

       [[b'1147', b'399'],
        [b'1280', b'541'],
        [b'bus', b'bus']],

       [[b'717', b'611'],
        [b'765', b'687'],
        [b'motorbike', b'motorbike']],

       [[b'1219', b'559'],
        [b'1280', b'658'],
        [b'rickshaw', b'rickshaw']],

       [[b'591', b'363'],
        [b'640', b'416'],
        [b'bus', b'bus']],

       [[b'505', b'572'],
        [b'571', b'664'],
        [b'rickshaw', b'rickshaw']],

       [[b'701', b'459'],
        [b'805', b'611'],
        [b'bus', b'bus']],

       [[b'592', b'440'],
        [b'700', b'587'],
        [b'bus', b'bus']]], dtype=object)]

This output contains bounding boxes and its class label in object type format. I receive around 50-60 outputs perfectly fine, then this error gets thrown

File "/home/ericedge/anaconda3/envs/skylark/lib/python3.6/site-packages/tritonclient/http/__init__.py", line 1926, in as_numpy
    self._buffer[start_index:end_index])
  File "/home/ericedge/anaconda3/envs/skylark/lib/python3.6/site-packages/tritonclient/utils/__init__.py", line 268, in deserialize_bytes_tensor
    sb = struct.unpack_from("<{}s".format(l), val_buf, offset)[0]
struct.error: unpack_from requires a buffer of at least 274435 bytes

This is the code snippet from where I’m getting the output:

inferenceOutputs = [response.as_numpy(output_name) for output_name in output_names]
for i in range(len(inferenceOutputs)):
                    for output in outputs[i]:
                        if output[0][0].decode() != '-1':
                            x1 = int(output[0][0])
                            y1 = int(output[0][1])
                            x2 = int(output[1][0])
                            y2 = int(output[1][1])
                            draw_boxes.add(img, x1, y1, x2, y2, output[2][0].decode())

where draw_boxes is a function to draw the bounding boxes alongwith its class label. How to solve this error, and why is it coming?

[sourabh-burnwal]

@dyastremsky I will try to come up with a python backend model which could reproduce the error and then raise a bug issue. Thanks to both of you for the help.

[dyastremsky]

I appreciate the additional information. The sample output encoded/decoded correctly when tested with the encoding/decoding functions. I appreciate the effort you both put into reproducing the results. However, if we cannot reproduce them, we cannot debug this issue. I’m happy you were able to find a workaround.

I am going to close this issue due to it being unreproducible. If that changes, please feel free to provide a way to reproduce the bug and we can look into it.