Detecting abrupt stream closing in C++ client library

Description Not exactly a bug, but I needed some clarifications regarding the C++ triton client API.

Triton Information What version of Triton are you using? 21.08

Are you using the Triton container or did you build it yourself? Container

To Reproduce

Pseudocode-ish example

InferenceServerGrpcClient::Create(&triton_client, url_, /*verbose*/ false);
triton_client->StartStream([&](tc::InferResult *result) {
   if (detect_end_of_stream(result)) atomic_global_bool.set(thread_done);
}

Here, the constructor for the inference server client internally creates a new thread that constantly receives data from the server https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/library/grpc_client.cc#L1530

Now the callback is invoked by the thread on receiving data, and the callback knows that it’s finished by identifying that the recived data corresponds to the end of a stream.

However, in the case that the server crashes for some reason, the thread simply exits without any indication whatsoever (// End loop if Read() returns false)

In my case, the callback sets a global boolean that the main thread polls to know when the data is ready to be used. But if the server abrubtly dies, the client library doesn’t inform that the thread has actually exited, which leads to a deadlock since the callback doesn’t get called.

So am I missing something obvious or should there be something like a FinishCallback invoked here? https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/library/grpc_client.cc#L1573

Expected behavior The library should have some mechanism to communicate connection disruption

[jbkyang-nvi]

The client does not such a functionality. Like I said ClientReaderWriter does not have API that can call isFinished() or onFinishedCallback() even though we can close it through Finish(). It seems like IsServerLive is something you can call to break out of the loop (as a part of the callback), or you can set a timeout on the AsyncStreamInfer() request to stop the deadlock

[git-bruh]

Alright, thanks!