Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Segmentation fault when delete server through C API

See original GitHub issue

Description I write a program following simple server.

After unload model, the server encounters segmentation fault when exits. Not happen every time, but with a high probability.

gdb log:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./triton_test -r /root/merged_zoo -t 1 -c 1000 -m my_tensorrt_model'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fab6f5fbd8c in nvidia::inferenceserver::RateLimiter::ModelContext::ContainsPendingRequests(int) ()
   from /opt/tritonserver/lib/libtritonserver.so
[Current thread is 1 (Thread 0x7fab58dfd000 (LWP 3442))]
(gdb) bt
#0  0x00007fab6f5fbd8c in nvidia::inferenceserver::RateLimiter::ModelContext::ContainsPendingRequests(int) ()
   from /opt/tritonserver/lib/libtritonserver.so
#1  0x00007fab6f5fc368 in nvidia::inferenceserver::RateLimiter::ModelInstanceContext::RequestRemoval() ()
   from /opt/tritonserver/lib/libtritonserver.so
#2  0x00007fab6f5fc3d6 in nvidia::inferenceserver::RateLimiter::ModelInstanceContext::WaitForRemoval() ()
   from /opt/tritonserver/lib/libtritonserver.so
#3  0x00007fab6f603561 in nvidia::inferenceserver::RateLimiter::UnregisterModel(nvidia::inferenceserver::TritonModel const*) ()
   from /opt/tritonserver/lib/libtritonserver.so
#4  0x00007fab6f6f4a03 in nvidia::inferenceserver::TritonModel::~TritonModel() () from /opt/tritonserver/lib/libtritonserver.so
#5  0x00007fab6f6f511d in nvidia::inferenceserver::TritonModel::~TritonModel() [clone .localalias] ()
   from /opt/tritonserver/lib/libtritonserver.so
#6  0x00007fab6f5989b7 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<nvidia::inferenceserver::(anonymous namespace)::BackendDeleter::operator()(nvidia::inferenceserver::InferenceBackend*)::{lambda()#1}> > >::_M_run() ()
   from /opt/tritonserver/lib/libtritonserver.so
#7  0x00007fab6f36ade4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007fab6f47f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00007fab6f1a9293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Triton Information ngc container r21.10

To Reproduce

a tensorrt model with 16 instance count, no ratelimiter config

Expected behavior Server exits normally.

Issue Analytics

State:
Created 2 years ago
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

handokucommented, Mar 17, 2022

Hi @handoku , may I know how do you build simple.cc? Thanks!

Generally, simple server willl be built alongside tritonserver binary.

If you want to build simple.cc only, just include relevant header files, compile it and link libtritonserver.so. The CMakeLists.txt can be helpful.

And I suggest you using tritonserver NGC container to avoid dependency problems.

0reactions

laochonlamcommented, Mar 17, 2022

Hi @handoku , may I know how do you build simple.cc? Thanks!

Generally, simple server willl be built alongside tritonserver binary.

If you want to build simple.cc only, just include relevant header files, compile it and link libtritonserver.so. The CMakeLists.txt can be helpful.

And I suggest you using tritonserver NGC container to avoid dependency problems.

Thank you @handoku so much for your speedy reply. That is super helpful!

Top Results From Across the Web

C++ delete segmentation fault

The problem arises when I execute this code twice. The first time, the function gets to delete(serv) alright. The second time, it causes ......

How To Avoid Or Remove A Segmentation Fault Core ...

To remove a segmentation fault core dumped error in c++ nachos, you can try one of the following methods: – Use a different...

Core Dump (Segmentation fault) in C/C++

When a piece of code tries to do read and write operation in a read only location in memory or freed block of...

Debugging Segmentation Faults

Segmentation faults are referred to as segfault, access violation or bus error. Hardware notifies the operating system about memory access violation. The OS ......

Identify what's causing segmentation faults (segfaults)

A segmentation fault (aka segfault) is a common condition that causes programs to crash; they are often associated with a file named core...