Enhance Log Topics for Pulsar Functions

Is your feature request related to a problem? Please describe. When using the logTopic feature of Pulsar Functions it works great, but seems a bit limited in it’s usage as only the message body is being sent to the logTopic as byte[].

Describe the solution you’d like Add an additional LogAppender.append method override that allows for the org.apache.logging.log4j.core.LogEvent object to be sent to the logTopic directly or allow the logTopic to except a custom schema and not just byte[]. This would allow consumers of the log topic to decipher what messages are of type INFO, WARN, ERROR, etc. The metadata in the org.apache.logging.log4j.core.LogEvent objects could be very valuable on the log consumption and analytics side.

src/main/java/org/apache/pulsar/functions/instance/LogAppender.java

Original

@Override
    public void append(LogEvent logEvent) {
        producer.sendAsync(logEvent.getMessage().getFormattedMessage().getBytes());
    }

Potential future maybe?

@Override
    public void append(LogEvent logEvent) {
        producer.sendAsync(logEvent);
    }

Describe alternatives you’ve considered I have considered creating a custom schema and topic and then just having my functions publish to that via the following and not using the built in Logger at all: context.newOutputMessage((Log) topicToWrite.get(), Schema.LOG).value(input).sendAsync();

I would rather use the functions built in Logger if at all possible.

[jerrypeng]

@one70six sounds good to me. We can even produce the logs with a schema e.g.

| level | classpath | message| |INFO | org.apache.logging.log4j.core.LogEvent| foo

[tisonkun]

Closed as stale. Please create a new issue if it’s still relevant to the maintained versions.