Need details on KFS V2 based GRPC method for inference

We have a few questions specific to KFS V2 based GRPC method for inference.

1. Is KFS V2 also meant to support tabular data based payload or only for tensor based ML/DL workload ?

2. As KFS V2 GRPC support only certain data types - https://github.com/kubeflow/kfserving/blob/master/docs/predict-api/v2/required_api.md#tensor-data-types. Is it recommended to extend KFS v2 GRPC protocol to support the other data types ?

In order to support additional data types, we are planning to take the following approach:

1. Extend the KFS v2 GRPC to support the additional data types - String, Array, Date and Timestamp etc as part of message “InferInputTensor” as shown below:

    message InferInputTensor
    {
        .......
        string datatype = 2; -> *** Accept additional data types - String, Array, Date and Timestamp etc ***
        .......
    }   

1. Underlying value would still be an array of bytes to align with KFS V2 GRPC protocol. Data for additional data types - String, Array, Date and Timestamp etc will get into “byte_contents”.

    message InferTensorContents
    {
        ....
        repeated bytes bytes_contents = 8;   -> Values for additional data types will be added here in bytes format
    }

Could you please comment on the approach, are there any other alternatives to support tabular data / other data types in KFS V2

[adriangonz]

Hey @vegoutha, in my view the best practice to encode tabular data is to treat each column as a separate “input head” (i.e. a separate entry on the inputs array of the payload). This would give you the best control over your inputs, in terms of shape, documentation (i.e. metadata / model signature), data types, etc.

Following your example, you could do something like:

{
	"inputs": [
		{
			"name": "id",
			"data": [1],
			"datatype": "int32"
		},
		{
			"name": "text",
			"data": ["hello world"],
			"datatype": "bytes"
		},
		{
			"name": "words",
			"data": [["hello", "world"]],
			"datatype": "bytes"
		},
		{
			"name": "features",
			"data": [[0, 1, 2, 3, 4]],
			"datatype": "fp32"
		},
	]
}

Leaving aside the richer datatype discussion (i.e. supporting a string), this is already supported by design in the V2 protocol.

[vegoutha]

@adriangonz How you are planning to support Spark runtime in ML server that includes tabular based payload as shown below ?

For example: Here is spark model schema format

[
  { "name": "id", "type": "LongType", "nullable": false },
  { "name": "text", "type": "StringType", "nullable": false},  
  { "name": "words", "type": "ArrayType", "nullable": false},
  { "name": "features","type":"VectorUDT","nullable": false}
]

How you are trying to accommodate the input data for above data types using KFS V2 GRPC ?

As mentioned in the MLServer design doc(https://docs.google.com/document/d/1C2uf4SaAtwLTlBCciOhvdiKQ2Eay4U72VxAD4bXe7iU/edit#heading=h.druw8i7fuflr), this case it will be passed as "content_type": "json" right?