Consumers not getting messages after broker restart

Although this project sounds very convenient, I can’t get it to work (at least with a consumer).

What I want to achieve is to start a consumer, then shut down the broker and restart it, and expecting the consumer to receive new messages.

I’m running RabbitMQ in Docker. Try this to reproduce the issue:

Run rabbitMQ in the foreground with

docker run -p 15672:15672 -p 5672:5672 --rm -it rabbitmq:3-management

Run this piece of java code (note I was trying to determine if there were types of exceptions I could add to the list of retryable/recoverable exceptions by using listeners, but nothing got logged).

import com.rabbitmq.client.*;
import net.jodah.lyra.ConnectionOptions;
import net.jodah.lyra.Connections;
import net.jodah.lyra.config.Config;
import net.jodah.lyra.config.RecoveryPolicies;
import net.jodah.lyra.config.RetryPolicies;
import net.jodah.lyra.event.DefaultChannelListener;
import net.jodah.lyra.event.DefaultConnectionListener;
import net.jodah.lyra.event.DefaultConsumerListener;
import net.jodah.lyra.util.Duration;

import java.io.IOException;
import java.util.concurrent.TimeoutException;

public class App {
    public static void main(String[] args) throws TimeoutException, IOException {

        Config config = new Config()
            .withRecoveryPolicy(
                RecoveryPolicies.recoverAlways()
                    .withBackoff(Duration.seconds(1), Duration.seconds(30))
            )
            .withRetryPolicy(
                RetryPolicies.retryAlways()
                    .withBackoff(Duration.seconds(1), Duration.seconds(30))
            )
            .withChannelListeners(new DefaultChannelListener() {
                @Override
                public void onRecoveryFailure(Channel channel, Throwable failure) {
                    System.out.println("CHANNEL RECOVERY FAIL");
                    failure.printStackTrace();
                }
            })
            .withConnectionListeners(new DefaultConnectionListener() {
                @Override
                public void onRecoveryFailure(Connection connection, Throwable failure) {
                    System.out.println("CONNECTION RECOVERY FAIL");
                    failure.printStackTrace();
                }
            })
            .withConsumerListeners(new DefaultConsumerListener() {
                @Override
                public void onRecoveryFailure(Consumer consumer, Channel channel, Throwable failure) {
                    System.out.println("CONSUMER RECOVERY FAIL");
                    failure.printStackTrace();
                }
            });

        ConnectionOptions options = new ConnectionOptions().withHost("localhost");
        Connection connection = Connections.create(options, config);

        Channel channel = connection.createChannel();
        channel.queueDeclare("foo", true, false, false, null);
        channel.basicConsume("foo", new FooConsumer(channel));
    }

    static class FooConsumer extends DefaultConsumer {
        public FooConsumer(Channel channel) {
            super(channel);
        }

        @Override
        public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
            System.out.println(new String(body));
            getChannel().basicAck(envelope.getDeliveryTag(), false);
        }
    }
}

In your browser go to the management console on localhost:15672. Go to the foo queue and publish some messages. You will see them being printed by the java program.

Now terminate the broker by hitting ctrl + c in the terminal where you started the docker container.

Just restart it with up + enter.

In the output log from RabbitMQ, you can tell that the java app reconnects:

=INFO REPORT==== 23-Feb-2017::17:57:37 ===
accepting AMQP connection <0.1105.0> (172.17.0.1:42122 -> 172.17.0.2:5672)

Also, in the management UI you can see that a new consumer appears at the foo queue, so lyra is definitely reconnecting and starting a new consumer behind the scenes.

However, when publishing new messages through the UI, nothing gets printed on the console where the java program runs. In the UI, you can see that the messages are being consumed, but not acknowledged.

Maybe it’s related to https://github.com/jhalterman/lyra/issues/74.

I hope you can find out what’s going on or point out what I’m doing wrong or misunderstanding. I’d really like to use this in my project to save myself the hassle of building my own error handling/reconnecting stuff.

Thanks!

[basert]

added a PR, not sure how to release a new version though, I think that is up to you 😃

[basert]

I was able to reproduce this issue by noticing that after the connection was restored by lyra, two consumers were on my queue. One was with this initial consumer tag, the other was with a new one.

I did some breakpoint debugging on the basicConsume method in com.rabbitmq.client.impl.ChannelN and the method gets called twice after the connection restore. This is due to changed behavior in rabbitmq > 4.0.0 where automatic recovery is enabled by default, resulting in a new consumer from lyra and the recovered by the rabbitmq library.

My solution was to disable this in the connection factory:

val connectionFactory = new com.rabbitmq.client.ConnectionFactory
connectionFactory.setAutomaticRecoveryEnabled(false)
val options = new ConnectionOptions(connectionFactory)