segfault at ... error 6 in libc-2.27.so

Description Docker container crashes with segfault.

Triton Information I’m using Triton container nvcr.io/nvidia/tritonserver:20.09-py3

To Reproduce Start Triton server with model with following configuration:

        name: 'Model'
        platform: 'tensorflow_savedmodel'
        input [
          {
            name: 'input_layer'
            data_type: TYPE_UINT8
            dims: [ -1, -1, -1, 3 ]
          }
        ]
        output [
          {
            name: 'tf_op_layer_Cast_1'
            data_type: TYPE_UINT8
            dims: [ -1, -1, -1, 3 ]
          }
        ]

        instance_group {
              kind: KIND_GPU
              count: 1
           }

Run two or more tasks constantly requesting inference using Triton client with cuda shared memory. One task may succeed, but another will cause segfault.

Expected behavior All tasks works correctly.

[vBLFTePebWNi6c]

@jbkyang-nvi I’ll give a try 21.06 and come back later. @CoderHam Here’s Triton Server log. Running tasks creates cuda shared memory regions and reuse them between inference requests, at the end script destroys created regions. Unfortunately I can’t provide code and models to reproduce this bug due to NDA.

[tanmayv25]

Glad it worked for you. If you see similar issues with our recent releases or have general suggestions on how to handle such cases better in triton, feel free to open another GitHub issue.