Integrating with Kafka-Node and similar Kafka clients

Part of our system relies on Kafka for message queues. I would like to be able to continue tracing transactions as they pass through Kafka via various producers and consumers, creating spans along the way. We are setting up kafka-node, which is a fairly common package, like this:

const kafka = require('kafka-node'),
    HighLevelProducer = kafka.HighLevelProducer,
    ConsumerGroup = kafka.ConsumerGroup,
    client = new kafka.KafkaClient({kafkaHost: kafkaConfig.kafkaHost}),
    producer = new HighLevelProducer(client),
    consumer = new ConsumerGroup(
        {
            kafkaHost: kafkaConfig.kafkaHost,
            groupId: kafkaConfig.groupId,
            fromOffset: 'earliest',
        },
        'os-events-raw'
    ),
    appEventsConsumer = new ConsumerGroup(
        {
            kafkaHost: kafkaConfig.kafkaHost,
            groupId: kafkaConfig.groupId,
            fromOffset: 'earliest',
        },
        'app-events'
    ),
    osEventsConsumer = new ConsumerGroup(
        {
            kafkaHost: kafkaConfig.kafkaHost,
            groupId: kafkaConfig.groupId,
            fromOffset: 'earliest',
        },
        'os-events'
    ),
    smcEventsConsumer = new ConsumerGroup(
        {
            kafkaHost: kafkaConfig.kafkaHost,
            groupId: kafkaConfig.groupId,
            fromOffset: 'earliest',
        },
        'smc-events'
    );

events get handled like this:

consumer.on('message', (kafkaMessage) => {
    let message =null;
    try {
        message = JSON.parse(kafkaMessage.value);
    } catch (e) {
        logger.error('failure parsing kafka message',{e,message})
        return;
    }
    if (!message) {
        logger.error('message is empty',{kafkaMessage})
        return;
    }

    let eventTransform = rawOsEventTransform(message);
    if (!eventTransform) {
        logger.error('XML parse error',{message})
        return;
    }

    producer.send([{
        topic: 'os-events',
        messages: [eventTransform]
    }], (err, data) => {
        if (err) {
            logger.error('kafka error', {err});
        }
    });
});

So, as you can see, a typical case is a consumer taking messages from a topic, transforming them and producing them to another topic.

If there is an example of how we can log this using the APM agent, then great. If not, I’d like to be able to do so.

[watson]

I’ve created a PR to update the types: #1001

[watson]

Oh, that’s probably because we didn’t update the TypeScript .d.ts file. That’s a mistake. I’ll make sure to fix it ASAP. Thanks for letting us know