It is not clear how to handle onsuccess and onerror results when using Kafka

The framework offers MessageChannel as a way to send the messages asynchronously/synchronously:

@Autowired
private Source source;

public void sayHello(String name) {
     source.output().send(MessageBuilder.withPayload(body).build());
}

However the MessageChannel API doesn’t allow the use of any type of callback to handle onsuccess and onerror results.

When a delivery fails I observe that there is a automatically registered LoggingProducerListener that prints the error:

public interface ProducerListener<K, V> {
    void onSuccess(String topic, Integer partition, K key, V value, RecordMetadata recordMetadata);
    void onError(String topic, Integer partition, K key, V value, Exception exception);
    boolean isInterestedInSuccess();
}

It would come in handy if we could specify a callback in the send() operation. Otherwise, how can we register a custom error handler per operation? How can we know that the operation was successful? Are we supposed to use directly the KafkaTemplate (KafkaTemplate.setProducerListener) ?

In any case it would be great if the doc were a little bit clearer in this regards.

Thanks!

[garyrussell]

Since version 1.3, you can use the error-channel-enabled producer property.

Example:

spring:
  cloud:
    stream:
      bindings:
        output:
          producer:
            error-channel-enabled: true
      kafka:
        bindings:
          output:
            producer:
              configuration:
                max:
                  block:
                    ms: 5000

and

@SpringBootApplication
@EnableBinding(Source.class)
public class Scstgh795Application {

	public static void main(String[] args) {
		SpringApplication.run(Scstgh795Application.class, args);
	}

	@Bean
	ApplicationRunner runner(MessageChannel output) {
		return args -> {
			System.out.println("Stop the brokers and hit enter");
			System.in.read();
			output.send(new GenericMessage<>("foo"));
		};
	}

	@ServiceActivator(inputChannel = "output.errors")
	public void errorHandler(ErrorMessage em) {
		System.out.println(em);
	}

}

(Use a service activator subscribed to the error channel).

and…

Stop the brokers and hit enter
...
2018-04-11 11:22:34.843 ERROR 69320 --- [           main] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='{102, 111, 111}' to topic output:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 5000 ms.

ErrorMessage [payload=org.springframework.integration.kafka.support.KafkaSendFailureException: nested exception is org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 5000 ms., failedMessage=GenericMessage [payload=byte[3], ... [record=ProducerRecord(topic=output, partition=null, ...

[garyrussell]

I updated the example to see how to get the results of successful sends, using a ProducerInterceptor:

spring:
  cloud:
    stream:
      bindings:
        output:
          producer:
            error-channel-enabled: true
      kafka:
        bindings:
          output:
            producer:
              configuration:
                max:
                  block:
                    ms: 5000
                interceptor:
                  classes: com.example.Scstgh795Application$MyInterceptor

@SpringBootApplication
@EnableBinding(Source.class)
public class Scstgh795Application {

	public static void main(String[] args) {
		SpringApplication.run(Scstgh795Application.class, args);
	}

	@Bean
	ApplicationRunner runner(MessageChannel output) {
		return args -> {
			System.out.println("Stop the brokers and hit enter");
			System.in.read();
			output.send(new GenericMessage<>("foo"));
			System.out.println("Start the brokers and hit enter");
			System.in.read();
			output.send(new GenericMessage<>("bar"));
			Thread.sleep(5_000);
		};
	}

	@ServiceActivator(inputChannel = "output.errors")
	public void errorHandler(ErrorMessage em) {
		System.out.println(em);
	}

	public static class MyInterceptor implements ProducerInterceptor<byte[], byte[]> {

		@Override
		public void configure(Map<String, ?> configs) {
		}

		@Override
		public ProducerRecord<byte[], byte[]> onSend(ProducerRecord<byte[], byte[]> record) {
			return record;
		}

		@Override
		public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
			if (metadata != null) {
				System.out.println(metadata);
			}
			if (exception != null) {
				System.out.println(exception);
			}
		}

		@Override
		public void close() {
		}

	}

}