1.0 API Feedback

This issue is to track community feedback on the new 1.0 API and document changes that are likely to be made before it’s release. We’ll continue to update this first comment to reflect open changes currently under consideration & progress.

For problems, please open another issue.

Comments / suggestions? Add them below!

Current changes under consideration for 1.0:

• add async serializers/deserializers (serdes that return Tasks) in addition to simple serde types.
• ProduceAsync - correctly implement a CancellationToken parameter.
• AdminClient.GetMetadata - remove this method and add methods analogous to the Java client.
• ListGroups - parse the group data and change the return type.
• Span<T> serializers.
• return types from AdminClient methods. Even though it’s not useful now, a return type from e.g. CreateTopicsAsync would allow for easier future extensibility.
• how should we communicate that a subscription didn’t match any topics?
• Exception types corresponding to more ErrorCodes.
• Investigate making Consumer<,> implement IEnumerable.
• pass message headers to deserializers
• Commit to throw CommitException, not TopicPartitionOffsetException.

Review TODOs:

• Add SCRAM-SHA-1 sasl mechanism.
• Avro serializers should have adaptive buffer sizing.
• Reconsider various strongly typed config enum names. in particular: Sasl_Plaintext, Sasl_Ssl, Roundrobin, Ssl.
• Ordering of configuration is important to librdkafka. this isn’t compatible with use of Dictionary<K,V>.
• Check that the float and double serdes are IEEE-754 compatible. Consider using IPAddress.HostToNetworkOrder.

[AndyPook]

Here we go again…

This “blocking” concern, just isn’t one. It is completely typical (especially with interop things). But we’ve had this discussion before. Suffice to say, doing async over sync is just fine. Of course, you don’t get the nice thread offloading by letting the IO system track it. But… this is far more acceptable than doing sync over async. Which is what trying to do both sync and async serdes in the same method forces us to do. So, if I use avro, or some other async serdes, I still do not get any benefit from Task/async! And we just get all the pain with zero gain.

I get and appreciate, that you’re trying to do “the right thing” and factor things in a compact way. But it leads it into an async pickle.

1. Pull out a base class with all but the Consume methods (conveniently last in the file). 2 descendants, pure sync and pure async (for the async serdes). We’ve discussed this before. But maybe with all the changes, this fits better now? Even is you only do one descendant that’s exactly like the current situation, it would allow us library users to our own mechanism and it side-steps the whole discussion 😃 or
2. A different way of keeping Consumer pure sync. Is to make it only use the sync serdes. But this only works if Deserializer also takes a SerializationContext (this would actually be useful if you’re using json, msgpack, others). Then you could have…

    public delegate T Deserializer<T>(ReadOnlySpan<byte> data, bool isNull, SerializationContext context);

    public static class Deserializers
    {
        public static Deserializer<T> Async<T>(IAsyncDeserializer<T> asyncSerializer) {
            return (data, isNull, context) =>
            {
                return asyncSerializer.DeserializeAsync(new ReadOnlyMemory<byte>(data.ToArray()), isNull, context)
                    .ConfigureAwait(false)
                    .GetAwaiter()
                    .GetResult();
            };
        }
...

So, we keep Consumer as pure sync and just wrap the async serde with the standard sync-over-async pattern. The Builder can deal with the wrapping and the Consumer is none the wiser.

Yes this is just a different method of hiding the async’ness of the async serdes, but, at least it vastly simplifies the Consumer.

[edenhill]

A boolean flag on ConsumeResult - IsPartitionEOF 👍