How to setup default topic config for consumer and producer with topic auto creation?

Description

Hello! At our team we use topics with auto creation on consumer subscription or producer publish. And we need to set default topic config for topic (at least setup cleanup policy and number of partitions). On some issues in this repo (f.e. https://github.com/confluentinc/confluent-kafka-dotnet/issues/861) i found some mentions about default.topic.config when configuration has type signature as IEnumerable<KeyValuePair<string, object>>. librdkafka have parameter in documentation to setup default topic config (default_topic_conf in https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). And now signature of config is more strongly typed with extra properties type signature as IEnumerable<KeyValuePair<string, string>> . I’ve tried to set ``default_topic_conf``` to config with dummy value and i’ve got:

System.ArgumentException : Property "default_topic_conf" must be set through dedicated .._set_..() function
   at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value)
   at Confluent.Kafka.Consumer`2.<>c__DisplayClass77_0.<.ctor>b__3(KeyValuePair`2 kvp)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Confluent.Kafka.Consumer`2..ctor(ConsumerBuilder`2 builder)
   at Confluent.Kafka.ConsumerBuilder`2.Build()

Seems like I need to call librdkafka api to set it up but required configHandle is encapsulated in Consumer ctor: https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/src/Confluent.Kafka/Consumer.cs#L676.

With default.topic.config:

System.InvalidOperationException : No such configuration property: "default.topic.config"
   at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value)
   at Confluent.Kafka.Consumer`2.<>c__DisplayClass77_0.<.ctor>b__3(KeyValuePair`2 kvp)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Confluent.Kafka.Consumer`2..ctor(ConsumerBuilder`2 builder)
   at Confluent.Kafka.ConsumerBuilder`2.Build()

What is a correct way to set default topic config? And if at the moment its not available do you have any plans to support it?

• Confluent.Kafka nuget version: 1.6.3
• Apache Kafka version: 0.11.
• Windows 10 Pro, Build 20H2

[CrazyRyDmi]

Thanks for explanation! Actually we use approach with explicit topic creation by wrapping Produce/Subscribe methods and calling something like EnsureTopicExistsAsync (which uses AdminClient) before corresponding methods with topic settings (server options like cleanup.policy). I think that its very convinient to use lazy topic initialization with custom config and without necessarity to use explicit checking about either topic already exists or not. Maybe this is not a bad idea to inlude feature like that into Confluent.Kafka in a future releases. What do you think about?

[CrazyRyDmi]

Great! Thanks for your answers!