thread safe Producer/Consumer/AdminClient Dispose methods

Calling Dispose on a Producer, Consumer or AdminClient whilst an instance method that makes use of the librdkafka handle is executing on another thread is not supported and may result in an access violation exception. Doing this is an anti-pattern, but we should provide better behavior if the user does do this (probably block Dispose until outstanding methods complete).

[mhowlett]

our documentation really needs improving yes. the tradeoffs are do you get transactions (for example) or better docs? you’re getting the former, but perhaps it should be the latter. we’ll get there. the only other place you need to be careful (that I can recall) is the log handler, since it can be called on any arbitrary librdkafka thread.

[pgrm]

@mhowlett sry in case I mixed up the terminology, yes my background tasks store the offset, the commit is done by keeping EnableAutoCommit on true

the background threads are actually slightly more sophisticated, I just didn’t want to get into it, as I thought that the issue most likely lies in the Dispose method, but just a quick recap - the records are processed by an Rx.Net-observable pipeline, guaranteed in order. There is one pipeline / subscribed topic + partition. The pipeline is basically equivalent to:

new Subject<ConsumeResult<string, string>>()
    .TakeWhile(_ => consumer.IsActive) // stop processing the pipeline when we can't store the offset anyway
    .Select(x => Observable.Create<ConsumeResult<string, string>>(async o =>
    {
        try
        {
            await consumeItemAsync(x);
            o.OnNext(x);
            o.OnCompleted();
        }
        catch (Exception ex)
        {
            o.OnError(ex);
        }
    }))
    // that whole thing above together with this concat guarantees synchronous in order processing
    .Concat()
    .TakeWhile(_ => consumer.IsActive) // double check if it wasn't disposed in the meanwhile
    .Do(consumer.StoreOffset) // the consumer has the locking implemented and should double check once more in case it wasn't really disposed in the meanwhile

and this part seems to work really smoothly.

We actually only have that problem in one service, maybe it’s because it’s handling the most messages, but maybe it’s because in this service handling a message can take quite a long time. Because the service is delivering messages to external parties, and their average response time is in the 10s of seconds, we have many concurrent pipelines active as waiting for an HTTP response doesn’t consume the thread from the pool.

Just trying to give you all the info I have - in case there is some weird edge case I don’t know about