Retry not executing code

I have run into a problem with the RetryTemplate. I’m building a client for jira (issue tracking) with feign and spring cloud stream. The issues that need to produced are send to the component over http, then the component enqueues the issues (buffering, our on premise jira is very unstable) on a queue using Spring Cloud Stream (Chelsea.SR2) and RabbitMQ binder. The same component also listens to the queue for sending the issues further to jira using feign. But the problem is that Spring Cloud Stream not always does the retry correctly. (Yes, sometimes it works, not always…) In the wrong flow you can see JiraClient isn’t invoked. (Yes we have multiple retries, jira client does retry 5 times, the message should retry for a long time, int max)

Expected flow:

[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (26ms)
[      main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
[      main] b.k.a.SuggestionsApplication             : Started SuggestionsApplication in 9.452 seconds (JVM running for 9.873)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 1000
[Consumer-1] o.s.retry.support.RetryTemplate          : Checking for rethrow: count=1
[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=1
[Consumer-1] Handler$$EnhancerBySpringCGLIB$$a467b7f4 : Dequeued message
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 2000
[Consumer-1] o.s.retry.support.RetryTemplate          : Checking for rethrow: count=2
[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=2
[Consumer-1] Handler$$EnhancerBySpringCGLIB$$a467b7f4 : Dequeued Message
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> RETRYING
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] ---> POST https://jira.be/jira/rest/api/2/issue HTTP/1.1
[Consumer-1] client.JiraClient      : [JiraClient#createIssue] <--- ERROR UnknownHostException: jira.be (0ms)
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 4000

Wrong flow:

[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=0
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 1000
[      main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
[      main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
[      main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
[      main] b.k.a.SuggestionsApplication             : Started SuggestionsApplication in 9.31 seconds (JVM running for 9.737)
[Consumer-1] o.s.retry.support.RetryTemplate          : Checking for rethrow: count=1
[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=1
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 2000
[Consumer-1] o.s.retry.support.RetryTemplate          : Checking for rethrow: count=2
[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=2
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 4000
[Consumer-1] o.s.retry.support.RetryTemplate          : Checking for rethrow: count=3
[Consumer-1] o.s.retry.support.RetryTemplate          : Retry: count=3
[Consumer-1] o.s.r.backoff.ExponentialBackOffPolicy   : Sleeping for 8000

Code:

    @StreamListener(JIRA_QUEUE)
    public void handleIssue(Issue issue) {
        if (issue != null) {
            log.info("Dequeued);
            jiraClient.createIssue(new IssueResource(issue));
        }
    }

Configuration:

spring.cloud.stream.bindings.jiraQueue:
  content-type: avro/bytes
  group: jiraConsumer
  requiredGroups: jiraConsumer
  consumer.maxAttempts: 2147483647

[artembilan]

OK. Look. You have this:

public interface IssueQueue {

    String JIRA_QUEUE = "issueQueue";

    @Input(JIRA_QUEUE)
    SubscribableChannel listenIssue();

    @Output(JIRA_QUEUE)
    MessageChannel pushIssue();
}

And seems for me you confuse the Spring Cloud Stream binding functionality with the same "issueQueue" for input and output. And in this case the @StreamListener is subscribed to the pushIssue MessageChannel proxy alongside with the SendingHandler to proceed to the target RabbitMQ destination.

Since we fail to send via Feign we get an exception on the @StreamListener and according round-robing logic move to the next subscriber - SendingHandler. The message is enqueued to the RabbitMQ and consumer starts to work. In this case the AmqpInboundChannelAdapter has the outputChannel as listenIssue .But voila! This one doesn’t have subscribers because our @StreamListener is on the pushIssue.

So, to fix your problem, consider to distinguish Producer and Consumer to different application or use different names for the @Input and @Output definitions meanwhile you can definitely bind them to the same target RabbitMQ destination via spring.cloud.stream.bindings. properties in the application.yml.

Meanwhile I think this is definitely bug and DispatchingStreamListenerMessageHandler must definitely subscribe to the binding channel marked with the @Input.

[JMesens]

It is easy to reproduce: https://github.com/JMesens/asyncHttp