I'm getting `ETIMEDOUT` and `authentication failed` on the 2nd topic only. How can I better troubleshoot this?

I’m having an issue subscribing a consumer to multiple topics. Usually the first topic or 2 will succeed nearly instantly, then it hits a wall. It will hang for several minutes then output 3 error messages. If I continue waiting it will throw the same 3 error messages.

I’m having a hard time figuring out if this is an issue on my end, kafkajs, or my company’s Kafka instances. I’m leaning towards the client side of things since other teams here consume happily with Java connectors.

NOTE: I redacted various things in code and the logs

Code:

(async () => {
	const topics = ['topic-a', 'topic-b', 'topic-c'];

	const kafka = new Kafka({
		clientId: 'some-client-id',
		brokers: ['some-broker.redacted.com:9093'],
		sasl: {/* REDACTED */},
	});

	const consumer = kafka.consumer({ groupId: 'some-group-id' });

	for (let topic of topics) {
		console.log(`Sub to ${topic}...`);
		await consumer.subscribe({ topic }); // hangs for several minutes on the second or 3rd topic
		console.log('...done');
	}

	// ... code for consumer.run below. It never makes it this far.
})();

Most of the time I get output like this:

Sub to topic-a...
...done
Sub to topic-b...
(Hangs here for several minutes...)
{"level":"ERROR","timestamp":"2019-10-14T21:25:12.210Z","logger":"kafkajs","message":"[Connection] Connection error: read ETIMEDOUT","broker":"some-broker-a.redacted.com:9093","clientId":"some-client-id","stack":"Error: read ETIMEDOUT\n    at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27)"}
{"level":"ERROR","timestamp":"2019-10-14T21:25:12.212Z","logger":"kafkajs","message":"[SCRAM512Authenticator] SASL SCRAM SHA512 authentication failed: Connection error: read ETIMEDOUT","broker":"some-broker-a.redacted.com:9093"}
{"level":"ERROR","timestamp":"2019-10-14T21:25:12.212Z","logger":"kafkajs","message":"[BrokerPool] KafkaJSSASLAuthenticationError: SASL SCRAM SHA512 authentication failed: Connection error: read ETIMEDOUT","retryCount":0,"retryTime":294,"stack":"KafkaJSSASLAuthenticationError: SASL SCRAM SHA512 authentication failed: Connection error: read ETIMEDOUT\n    at SCRAM512Authenticator.authenticate (/path-to-project/node_modules/kafkajs/src/broker/saslAuthenticator/scram.js:157:21)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"}

If I let it sit it will eventually throw the same 3 error messages.

[tulios]

consumer.subscribe will first refresh the current metadata if it is necessary, then it will randomly select a broker it knows, and it will request metadata for the specific topic you provided.

I noticed that you are getting the error SASL SCRAM SHA512 authentication failed, and your example is using port 9093, which usually only accepts SSL connections but no SASL. If you provide SASL configurations it the client will attempt to authenticate, but since the port doesn’t accept SASL, it might never reply, leading to a timeout.

[JaapRood]

If this was the problem, wouldn’t it NEVER work? I see now that I wasn’t super clear on this, but it usually fails but does occasionally work.

A broker known to the client is randomly selected to request the metadata for the specific topic. If that broker also doesn’t hold the partitions you’re trying to access, it’s conceivable how that particular broker just isn’t used.