bug: Trace output for `BYTES` has invalid JSON encoding

Description The trace file being written to disc doesn’t not contain validate json due to the addition quotes being include in the output of BYTES (string) inputs. See below the text "male" is being output with ""make"".

Triton Information What version of Triton are you using?

• nvcr.io/nvidia/tritonserver:22.05-pyt-python-py3

Are you using the Triton container or did you build it yourself?

• triton container

To Reproduce Steps to reproduce the behavior.

Following is the example input payload that I sent to the model:

payload = {
    "inputs": [
        {"name": "Sex", "shape": [1], "datatype": "BYTES", "data": ["male"] },
        {"name": "Age", "shape": [1], "datatype": "FP32", "data": [34.5] },
    ]
}

And a sample from the trace output.

{"id":1,"activity":"TENSOR_QUEUE_INPUT","tensor":{"name":"Sex","data":""male"","shape":"1","dtype":"BYTES"},
{"name":"Age","data":"34.5","shape":"1","dtype":"FP32"}

I am using an ensemble model with the input/outputs as show below:

name: "titanic_ensemble"
platform: "ensemble"
max_batch_size: 0 # Dynamic batch not supported
input [
  {
    name: "Pclass"
    data_type: TYPE_STRING
    dims: [ 1 ]
  },
  {
    name: "Sex"
    data_type: TYPE_STRING
    dims: [ 1 ]
  },
  {
    name: "Age"
    data_type: TYPE_FP32
    dims: [ 1 ]
  },
  {
    name: "SibSp"
    data_type: TYPE_FP32
    dims: [ 1 ]
  },
  {
    name: "Parch"
    data_type: TYPE_FP32
    dims: [ 1 ]
  },
  {
    name: "Fare"
    data_type: TYPE_FP32
    dims: [ 1 ]
  },
  {
    name: "Embarked"
    data_type: TYPE_STRING
    dims: [ 1 ]
  }
]
output [
  {
    name: "Survived_predictions"
    data_type: TYPE_STRING
    dims: [ 1 ]
  },
  {
    name: "Survived_probabilities"
    data_type: TYPE_FP32
    dims: [ 1, 3 ]
  }
]
...

Expected behavior I would expect to be able to parse the json file, but attempting to pipe to JQ fails eg:

cat trace.json | jq

It seems that additional quotes are added for the BYTES type which is in addition to the quote being add a the start and end of the data value.

A potential fix could be to not quote all outputs, which would mean the 34.5 value is returned as a floating point in json that matches the input. Otherwise, quotes in strings should get escaped.

I notice that each file is an actual json array. I wonder if tracer could be simpler and more scalable to just create a JSON lines output, so that you append to the buffer and can flush without having to start and end the trace file with an array [ ] ?

[kthui]

This fix is scheduled for 22.08 CC @GuanLuo @nv-kmcgill53

[brightsparc]

I’ve found one further bug in the trace_summary implement and proposed a fix @kthui