ProduceAsync too slow

Description

I’m working on a software to consume a stream from a Socket.IO source and putting into Kafka. The messages are about 100bytes each and I receive about 20000 messages per second from Socket.IO. For each message I call ProduceAsync().Result but this method slows down to 1 message per second.

Why is this happening? How can I resolve?

Other observations

• I need a strict order of the messages so I cannot remove the .Result and use the async method.
• I’ve checked the time span of both the serialize and deserialize functions (I use custom functions) and both are above 50ms
• Till some hours ago I was working with RC2 and the bug was still the same, I’ve updated the version hoping it was fixed
• The stream source (the one from Socket.IO) works like a charm if I don’t produce it into kafka
• I’ve found many other users experiencing a very similar problem with the consumer #619, #573 , #554
• @mhowlett here states that “maximum sustainable consume throughput is about 150k msg/s for 100 byte messages” I hope for the producer is about the same
• I’ve found other users with the same problem in #501 , #495, #471

Checklist

Please provide the following information:

• Confluent.Kafka nuget version: 1.0.0-BETA
• Apache Kafka version: 2.11
• Client configuration: SocketBlockingMaxMs = 1
• Operating system: both Windows and Docker Linux
• Provide logs (with “debug” : “…” as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[edenhill]

Synchronous produce will be slow, effectively limited by the broker round-trip and processing time.

You might want to try out the new idempotent producer support which adds in-order guarantee. Enable with enable.idempotence=true on the producer.

Do note that this feature is still in beta state.

[sean-buchanan]

Clearly that’s expensive! That’s what we were doing wrong. We just need to keep the publisher alive and reuse it for each call.

Thanks for the help!

Sean Buchanan

---

From: Matt Howlett [e-mail] Sent: Friday, October 5, 2018 2:21:44 PM To: confluentinc/confluent-kafka-dotnet Cc: sean-buchanan; Mention Subject: Re: [confluentinc/confluent-kafka-dotnet] ProduceAsync too slow (#630)

the only 1s delay I see there is after you shut down the producer. are you creating a new producer for every produce call? that’s extremely expensive.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/confluentinc/confluent-kafka-dotnet/issues/630#issuecomment-427487681, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUFmzJECAOss9ddm3Qff4eSxxZCMKxV8ks5uh79YgaJpZM4W5LA8.