Unhandled Exception: Confluent.Kafka.KafkaException: Disconnected after ms in state up

Description

Sometimes I got error:

Unhandled Exception: Confluent.Kafka.KafkaException: 10.244.3.12:9092/1: Disconnected (after 1438771ms in state UP) at Common.EventBus.KafkaProducer.<.ctor>b__4_1(IProducer2 _, Error errorHandler) at Confluent.Kafka.Impl.NativeMethods.NativeMethods_Debian9.rd_kafka_poll(IntPtr rk, IntPtr timeout_ms) at Confluent.Kafka.Producer2.<>c__DisplayClass24_0.<StartPollTask>b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

ctor of my KafkaProducer class

public KafkaProducer(EventBusOptions options, ILogger<KafkaProducer> logger)
{
	_options = options ?? throw new ArgumentException(nameof(options));
	_logger = logger ?? throw new ArgumentNullException(nameof(logger));

	var producerConfig = new ProducerConfig
	{ 
		BootstrapServers = options.ConnectionString, 
		EnableIdempotence = true,
		CompressionType = CompressionType.Snappy
	};

	_producer = new ProducerBuilder<string, string>(producerConfig)
		.SetLogHandler((_, logMessage) => _logger.LogInformation(logMessage.Message))
		.SetErrorHandler((_, errorHandler) =>
		{
			_logger.LogError(errorHandler.Reason);
			throw new KafkaException(errorHandler);
		})
		.Build();
}

How to reproduce

After sending message like

public async Task Send(IntegrationEvent eventData)
{
	var jsonMessage = JsonConvert.SerializeObject(eventData, new JsonSerializerSettings
	{
		ContractResolver = new CamelCasePropertyNamesContractResolver()
	});

	var message = new Message<string, string>
	{
		Value = jsonMessage,
		Headers = new Headers
		{
			{"eventType", Encoding.UTF8.GetBytes(eventData.EventType.ToLowerInvariant()) }
		}
	};

	_producer.Produce(topicName, message, report =>
	{
		if (report.Error.IsError)
		{
			_logger.LogError(report.Error.Reason);
		}
		else
		{
			_logger.LogInformation("Event message '{EventType}' with Id '{MessageId}' sent to topic '{TopicName}'", eventData.EventType, eventData.Id.ToString(), report.TopicPartitionOffset);
		}
	});
}

Log

Mar 4, 2020 @ 12:07:14.214	Event message ‘“con_integration”’ with Id ‘“c846365a-dcc7-4a48-b6f3-5116aeb72b96”’ sent to topic ‘“dev_con_integration [[11]] @164”’
Mar 4, 2020 @ 12:17:14.439	10.244.2.165:9092/0: Disconnected (after 992313ms in state UP)
–	–

Checklist

• Confluent.Kafka nuget version 1.3.0
• Apache Kafka image: confluentinc/cp-kafka:5.0.1
• Operating system: Linux
• Critical issue.

[mhowlett]

that error is not fatal.

[mhowlett]

if you do get a fatal error, every call to every method will also give you an exception. if you don’t it’s fine to keep using the producer. the name ‘error’ here is unfortunate and historical reasons - it’s confusing, yes.