Stateful model example for the python backend

Is your feature request related to a problem? Please describe. I am trying to make a stateful model in Python backend, but I couldn’t find any example to do that.

Describe the solution you’d like It would be great if someone can provide an example with a simple python backend, and ideally with the client code to start with.

More specifically, I wonder how we can access the sequence_id, sequence_start, sequence_end and other control signal information from the request object.

[zzk0]

I was wrong. sorry 😢

I found it! You can get sequence_id and correlation_id

https://github.com/triton-inference-server/python_backend/blob/main/src/pb_stub.cc#L661

EDIT: SO, Python Backend supports sequence batching. sorry for my misleading answer. 😰

[oeway]

Hi, I confirm that the python backend supports sequences. Would still be nice to add some documentation for accessing sequence related information.

In the config.pbtxt I added:

sequence_batching {
  max_sequence_idle_microseconds: 30000000
  direct {}
  control_input [
    {
      name: "START"
      control [
        {
          kind: CONTROL_SEQUENCE_START
          fp32_false_true: [ 0, 1 ]
        }
      ]
    },
    {
      name: "END"
      control [
        {
          kind: CONTROL_SEQUENCE_END
          fp32_false_true: [ 0, 1 ]
        }
      ]
    }
  ]
}

And in the python model, I can access them by:

sequence_id = request.correlation_id()
start_flag = pb_utils.get_input_tensor_by_name(request, "START").as_numpy()[0]
end_flag = pb_utils.get_input_tensor_by_name(request, "END").as_numpy()[0]

Will close this issue for now.