Consumer streams stop consuming

Hi, I’m using the consumer stream in an ETL service. Currently there are around 60 active consumers and everything works great. Once in a couple of days I notice that one of the streams stop consuming data and I need to manually restart it, yet there isn’t any error being emitted regarding this. I Use manual commits and don’t wait for the commit callbacks, not sure if that might be related to the issue. I currently use the stream.on('error', cb) for error handling. Is there a better way to get some more data for debugging? Thanks.

Here is the consumer configuration:

const hosts = config.kafka.hosts.join(',');

  const globalConfig = {
    'metadata.broker.list': hosts,
    'group.id': groupId,
    'enable.auto.commit': false,
    'offset.store.method': 'broker',
    'partition.assignment.strategy': 'roundrobin',
    'topic.metadata.refresh.interval.ms': 30000,
    'queued.max.messages.kbytes': 10000
  };

  const topicConfig = {
    'auto.offset.reset': 'latest'
  };

  const stream = new Kafka.createReadStream(globalConfig, topicConfig, {
    topics: [topicsRegex],
    waitInterval: 0,
    fetchSize: 1000
  });

  stream.on('error', (err) => {
    if (stream.consumer && stream.consumer._isDisconnecting) {
      logger.error('Error in kafka: Disconnecting in progress');
      return;
    }

    logger.error('Error in kafka ', {
      error_message: err.stack,
      error_code: err.code
    });
  });

[bobrowadam]

@webmakersteve @csaroff We now use much less consumers per CPU core and also made sure we gracefully close the consumers on restart and things works just fine ever since.

[bobrowadam]

Ok thanks, I’ll give it a try.