Implicit stream subscriptions seem to stop being called after some time

We’re seeing a grain implicitly subscribed to multiple streams suddenly stop having it’s OnNextAsync() method called. Other explicitly subscribed grains to one of the streams continue to work as expected. Other implicitly subscribed grains continue to work as expected.

Orleans Version:

Microsoft Orleans Client => 2.0.3
Microsoft Orleans Core => 2.0.3
Microsoft Orleans Core_Legacy => 2.0.3
Microsoft Orleans Core_Abstractions => 2.0.0
Microsoft Orleans OrleansCodeGenerator_Build => 2.0.3
Microsoft Orleans OrleansProviders => 2.0.4
Microsoft Orleans Runtime => 2.0.4
Microsoft Orleans Server => 2.0.4
Microsoft Orleans Runtime_Legacy => 2.0.4
Microsoft Orleans TestingHost => 2.0.4

I have a grain class defined similar to:

[StorageProvider(ProviderName = DocumentPersistence.ServiceLifetimeRedisStorage)]
[ImplicitStreamSubscription(DocumentStreams.DocumentA)]
[ImplicitStreamSubscription(DocumentStreams.DocumentB)]
public class MyGrain : Grain<MyGrainState>,
    IMyGrain,
    IAsyncObserver<IEnumerable<DocumentA>>,
    IAsyncObserver<IEnumerable<DocumentB>>
{
    public override async Task OnActivateAsync()
    {
        var id = this.GetPrimaryKey();

        var provider = GetStreamProvider(DocumentStreams.StreamProvider);

        var documentAStream = provider.GetDocumentStream<DocumentA>(id);
        await documentAStream.SubscribeAsync(this);

        var documentBStream = provider.GetDocumentStream<DocumentB>(id);
        await documentBStream.SubscribeAsync(this);
    }

    public async Task HandleDocument(DocumentA change)
    {
        // ...
    }

    public async Task HandleDocument(DocumentB change)
    {
        // ...
    }

    public async Task OnNextAsync(IEnumerable<DocumentA> items, StreamSequenceToken token = null)
    {
        var tasks = new List<Task>();
        foreach (var document in items)
        {
            tasks.Add(HandleDocument(document));
        }

        await Task.WhenAll(tasks);
    }

    public async Task OnNextAsync(IEnumerable<DocumentB> items, StreamSequenceToken token = null)
    {
        var tasks = new List<Task>();
        foreach (var document in items)
        {
            tasks.Add(HandleDocument(document));
        }

        await Task.WhenAll(tasks);
    }

    public Task OnCompletedAsync() => Task.CompletedTask;

    public Task OnErrorAsync(Exception ex)
    {
        GetSerilog().Error(ex, "An exception occurred while receiving stream data");
        return Task.CompletedTask;
    }
}

Most of the time all is well and OnActivatedAsync() followed by OnNextAsync() calls fire as one would expect. However, left running over night, we no longer receive OnNextAsync() calls when expected - none to be exact. Other grains that are subsequently, explicitly subscribed to the streams continue to work.

I should point out that we are using a redis stream and storage provider; however, all other stream subscriptions (both explicit and implicit) are working fine. It’s, seemingly, only this grain with two implicit subscriptions that is no longer receiving stream messages.

Some things I’ve tried to suss out what is actually going on:

• Tested other stream flows
• GrainCollectionOptions.CollectionAge = TimeSpan.FromMinutes(1) to force grain collection. I was able to repro the issue but only after still letting it run over night. Once in this state, waiting a number of minutes, I was never able to get OnActivatedAsync() to fire in the grain. This seems to imply the grain was either inaccessible due to some deadlock or busy waiting or that the grain itself was never activated and sent the message. Given that the grain is relatively simple and devoid of any locks, I am suspicious that it’s the latter.

[shlomiw]

@berdon - it would be helpful if you’d share in general what was the issue. Maybe we could learn from it as well. Thanks in advance

[berdon]

An unfortunate return instead of continue in the OnNextAsync. So, just a trivial programmer error. 😦