Lock contention on high load

From a comment on issue #177

Since it only happens in specific load conditions, it’s hard for me to reproduce it right away, but looking at the logs, I think it’s from a producer.send call. You can find the > following snippet on how I’m initializing the producer and how I’m calling the send function.

// Incoming data ... Can be considered something like this:
for (let i = 0; i < 100000; i++)
    queueMessage(message, 'specific-topic');

// Producer chunk ...
const kafkaClient = new Kafka({
    clientId: `SomeID`,
    brokers: ['10.240.0.6:9092', '10.240.0.7:9092', '10.240.0.20:9092'],
    connectionTimeout: 5000,
    requestTimeout: 60000,
    maxInFlightRequests: 200,
    retry: {
        initialRetryTime: 1000,
        retries: 5
    }
});

const producer = kafkaClient.producer();

async function queueMessage(message, topic, retries = 0) {
    try {
        await producer.send({
            topic,
            // TODO -> KafkaJS probable bug? key=null should be present
            messages: [{key: null, value: JSON.stringify(message)}],
            acks: 1,
            compression: kafkaUtils.SNAPPY
        });
    } catch (e) {
        if (retries < 2)
            setTimeout(() => queueMessage(message, topic, retries + 1), 2000 * (retries + 1));
    }
}

I’m not using any Promise.all, and the flow of incoming messages is a stream of small messages, so I can’t actually batch them together and send them, it’s more like streaming a lot of > messages in realtime, for a messaging application.

And I’m using KafkaJS 1.4.7.

@AlirezaSadeghi I moved the investigation to a new issue

[tulios]

@AlirezaSadeghi 1.5.2 was released with the fix.

[AlirezaSadeghi]

That would be great! I’ll be all out helping fix this issue because it’s really reducing the reliability of the messaging mechanism.

About the consumers and producers, yes, they are in the same process, there’s basically one consumer and one producer in the same process. (i.e app.js starts up the producer first, and then starts the consumer, consumer starts consuming messages and sending them, and then messages start coming back to the process and here, producer starts sending them back to a kafka topic)

And initially, I was using 1.4.7, I’m giving 1.5.0 a try now to see how it behaves. I’ll also increase those two again, but they are already 60 seconds or so.