Tensorflow model does not work on inference

Description I have a Tensorflow model, saved as ‘savedmodel’. Trying to deploy it with triton server and tensorflow_savedmodel platform. Triton server has loaded, my model is loaded, but when I try to use it and send a test image from client - I get an error:

    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INVALID_ARGUMENT] unexpected inference output 'output' for model 'efnet'

I have no Idea how to troubleshoot this, can’t find anything in documentation. Please give me some hints on where to look for an issue

Triton Information tritonserver2.19.0-jetpack4.6.1

Are you using the Triton container or did you build it yourself? built

To Reproduce config.pbtxt:

name: "efnet"
platform: "tensorflow_savedmodel"
max_batch_size: 1
input [
  {
    name: "input"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
  }
]
output [
  {
    name: "output"
    data_type: TYPE_FP32
    dims: [ 5 ]
  }
]

client:

import tritonclient.grpc as grpcclient
import numpy as np
import cv2
import sys


class Efnet_grpc():
    def __init__(self,
                 url="localhost:8001",
                 model_name="efnet",
                 input_width=224,
                 input_height=224,
                 model_version="",
                 verbose=False, conf_thresh=0.7) -> None:

        self.model_name = model_name
        self.input_width = input_width
        self.input_height = input_height
        self.batch_size = 1
        self.conf_thresh = conf_thresh
        self.input_shape = [self.batch_size, 3, self.input_height, self.input_width]
        self.input_name = 'input'
        self.output_name = 'output'
        self.output_size = 5
        self.triton_client = None
        self.init_triton_client(url)
        print('test_pred')
        self.test_predict()
        print('test_pred_done')


    def init_triton_client(self, url):
        try:
            triton_client = grpcclient.InferenceServerClient(
                url=url,
                verbose=False,
                ssl=False,
            )
        except Exception as e:
            print("channel creation failed: " + str(e))
            sys.exit()
        self.triton_client = triton_client


    def test_predict(self):
        input_images = np.zeros((*self.input_shape,), dtype=np.float32)
        _ = self.predict(input_images)


    def predict(self, input_images):
        inputs = []
        outputs = []

        inputs.append(grpcclient.InferInput(self.input_name, [*input_images.shape], "FP32"))
        # Initialize the data
        inputs[-1].set_data_from_numpy(input_images)
        outputs.append(grpcclient.InferRequestedOutput(self.output_name))

        # Test with outputs
        results = self.triton_client.infer(
            model_name=self.model_name,
            inputs=inputs,
            outputs=outputs)

        # Get the output arrays from the results
        return results.as_numpy(self.output_name)


    def preprocessing(self, image):
        image = cv2.resize(image, dsize=(self.input_width, self.input_height), interpolation=cv2.INTER_CUBIC)
        final_image = np.expand_dims(image, axis=0)
        return final_image


    def classifier_pred(self, image):
        proc_image = self.preprocessing(image)
        pred = self.predict(proc_image)
        print(pred)

As you can see, I have ‘input’ and ‘output’ both in client and config file. What’s wrong here?

[dyastremsky]

Those numbers can be int8, but it depends on what the model’s input types are. If a TensorFlow input was generated to be DT_FLOAT, then Triton will expect it to be FP32.

[dyastremsky]

Fantastic, any time!