SignalR streaming client can hang indefinitely if it does not receive a Completion on the streaming Channel

I have a pretty generic SignalR server-to-client streaming implementation with a .NET client. The hub exposes a streaming method like so:

public ChannelReader<byte[]> Retrieve(Guid id, CancellationToken cancellationToken)
{
    var channel = Channel.CreateBounded<byte[]>(_limit);
    _ = WriteItemsAsync(channel.Writer, id, cancellationToken);
    return channel.Reader;
}

private async Task WriteItemsAsync(ChannelWriter<byte[]> writer, Guid id, CancellationToken cancellationToken)
{
    Exception localException = null;
    try
    {
        //loop and write to the ChannelWriter until finished
    }
    catch (Exception ex)
    {
        localException = ex;
    }
    finally
    {
        writer.Complete(localException);
    }
}

and the client consumes the stream like this:

var channel = await hubConnection.StreamAsChannelAsync<byte[]>("Retrieve", _guid, cancellationTokenSource.Token);

while (await channel.WaitToReadAsync())
{
    while (channel.TryRead(out var data))
    {
        //handle data
    }
}

The client feeds the stream into a data flow pipeline and the server will call Complete on the Channel when that particular stream is done. The problem is that very seldomly (once every few thousand streaming sessions) the entire pipeline will hang completely, with the effect being that the client thinks it is still processing a stream and won’t request any further streams from the server, causing the whole pipeline to come to a halt.

My server logs will show that the server has called Complete on the ChannelWriter in the hub, but that the client is still blocking at await channel.WaitToReadAsync() as if it has not seen the Complete message. Because it has only happened a handful of times, I don’t have steps to reproduce it yet and I am not 100% positive SignalR is the culprit, but an answer to the following would help me narrow it down.

So my question: when the hub calls Complete on a streaming Channel, is that Complete message guaranteed to make it to the client, or is it possible that the message doesn’t make it in a rare case and the client blocks forever on a stream that is effectively closed? Obviously normal SignalR messages are “fire and forget”, but do the completion messages that SignalR sends under the hood for streams follow the same rules?

My workaround right now is to use the CancelAfter method of a CancellationTokenSource to allow me to timeout the streaming client if it hasn’t received any messages on the open stream within a given time span. I’ll be able to share more data when the next hang occurs.

[davidfowl]

Right it should never hang forever. It’ll either send the complete message or disconnect. Either way, a hang would be a bug in the client

[koopaking3]

Thanks again! I’ll close the issue and reopen later if I’m able to get a dump and there is a related issue.