segment fault when running tritonserver which is built from source

Description I successfully built triton server from source but got segment fault when running it.

$ tritonserver --help
[1]    2111498 segmentation fault (core dumped)  tritonserver --help

Already tried r21.06 and r21.07, both give the same error.

Just want to know what to do next.

Triton Information r21.07 and r21.06

Are you using the Triton container or did you build it yourself?

Build triton server by my self with the following command:

python build.py --no-container-build --build-dir `pwd`/builddir --enable-logging --enable-stats --enable-tracing --enable-metrics --enable-gpu-metrics --enable-gpu --cmake-dir=`pwd`/build --endpoint=http --endpoint=grpc --backend=ensemble

Host info:

• Ubuntu 20.04
• CUDA 11.0
• CUDNN 8.0.5

[nicklhy]

@tanmayv25 Thanks for your replies. I just tried the gdb debug and the backtrace is as below

(gdb) r
Starting program: /home/lhy/Documents/Lib/triton-server/builddir/opt/tritonserver/bin/tritonserver 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00005555558d4f3f in google::protobuf::internal::(anonymous namespace)::InitSCC_DFS(google::protobuf::internal::SCCInfoBase*) ()
#2  0x00005555558d5aa0 in google::protobuf::internal::InitSCCImpl(google::protobuf::internal::SCCInfoBase*) ()
#3  0x00007ffff6d445c5 in InitDefaults_dcgm_2eproto() () from /lib/x86_64-linux-gnu/libdcgm.so.2
#4  0x00007ffff6d8f551 in ?? () from /lib/x86_64-linux-gnu/libdcgm.so.2
#5  0x00007ffff6d8f639 in google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable*, void (* const*)(), int) () from /lib/x86_64-linux-gnu/libdcgm.so.2
#6  0x00007ffff7fe0b8a in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff7fe0c91 in ?? () from /lib64/ld-linux-x86-64.so.2
#8  0x00007ffff7fd013a in ?? () from /lib64/ld-linux-x86-64.so.2
#9  0x0000000000000001 in ?? ()
#10 0x00007fffffffcec4 in ?? ()
#11 0x0000000000000000 in ?? ()

About the version requirements in nvidia support matrix, I found the latest releases 21.04-21.07 require CUDA 11.3 which is too new for our cluster. And that’s exactly the reason why I am trying to build triton-server myself. ( We want to try the latest triton-server in our existing cluster.)

[tanmayv25]

We were always building Triton with the latest DCGM and recently fixed upon a version: https://github.com/triton-inference-server/server/pull/3140

This should be no longer an issue from next releases. build.py will install the dcgm from the one specified in VERSION_MAP.