[BUG] com.microsoft.azure.servicebus.amqp.AmqpException: The link 'G8:15665319:Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' is force detached. Code: consumer(link4236931)

I have created a small demo application to recreate the issue we are running into during production.

We have ServiceBusExceptionDemo running that is listening to a queue demoinboxqueue. It can just recieve message fine. The problem occurs when we change the Lock duration of the queue while the demo application is running. I notice that an exception is rightfully thrown and that the library is then trying to recreate the receive link: primitives.CoreMessageReceiver - Receive link 'Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' to 'demoinboxqueue', sessionId 'null' will be reopened after 'PT1S'

Bug: Connection/Reciever link is not correctly restored. When there are new messages on the queue the demo application is unable to receive them. (after the Lock duration via properties screen of queue in azure portal is changed)

import com.microsoft.azure.servicebus.*;
import com.microsoft.azure.servicebus.primitives.ConnectionStringBuilder;
import com.microsoft.azure.servicebus.primitives.ServiceBusException;
import java.time.Duration;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class ServiceBusExceptionDemo implements IMessageHandler {

    public static void main(String[] args) {
        ServiceBusExceptionDemo demo = new ServiceBusExceptionDemo();
        try {
            demo.setupQueueClient();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    private ExecutorService executorService = Executors.newSingleThreadExecutor();
    private String connectionString = "Endpoint=sb://demoExampleSB.servicebus.windows.net/;SharedAccessKeyName=Listen-demoinboxqueue;SharedAccessKey=1L2uJtU2OC7Ze8ojcIuzS783pHcN75ooCHpQ7BSCBYE=;EntityPath=demoinboxqueue";
    private IQueueClient client;

    public void setupQueueClient() throws InterruptedException {
        try {
            client = new QueueClient(new ConnectionStringBuilder(connectionString), ReceiveMode.PEEKLOCK);
            client.registerMessageHandler(
                this,
                new MessageHandlerOptions(5,
                    true,
                    Duration.ofMinutes(5L),
                    Duration.ofMinutes(60)
                ), executorService
            );

        } catch (ServiceBusException | InterruptedException e) {
            e.printStackTrace();
        }
    }

    @Override
    public CompletableFuture<Void> onMessageAsync(IMessage message) {
        System.out.println( "onMessageAsync: " + new String(message.getMessageBody().getBinaryData().get(0)));
        return CompletableFuture.completedFuture(null);
    }

    @Override
    public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
        throwable.printStackTrace();
    }
}

I get the following error:

2019-11-11 15:01:53.933 INFO   us.primitives.MessagingFactory - Creating messaging factory from connection string 'Endpoint=sb://demoExampleSB.servicebus.windows.net/;EntityPath=demoinboxqueue;'
2019-11-11 15:01:53.963 INFO   us.primitives.MessagingFactory - Creating messaging factory from namespace endpoint uri 'sb://demoExampleSB.servicebus.windows.net/'
2019-11-11 15:01:53.977 INFO   us.primitives.MessagingFactory - Creating and starting reactor
2019-11-11 15:01:54.047 INFO   us.primitives.MessagingFactory - Started reactor
2019-11-11 15:01:54.047 INFO   us.primitives.MessagingFactory - starting reactor instance.
2019-11-11 15:01:54.064 INFO   us.primitives.MessagingFactory - Creating connection to host 'demoExampleSB.servicebus.windows.net:5671'
2019-11-11 15:01:55.087 INFO   us.primitives.MessagingFactory - MessagingFactory opened.
2019-11-11 15:01:55.089 INFO   us.primitives.MessagingFactory - Connection opened to host.
2019-11-11 15:01:55.095 INFO   us.primitives.MessagingFactory - Creating CBS link to $cbs
2019-11-11 15:01:55.097 INFO   t.azure.servicebus.QueueClient - Created queue client to connection string 'Endpoint=sb://demoExampleSB.servicebus.windows.net/;EntityPath=demoinboxqueue;'
2019-11-11 15:01:55.098 INFO   rvicebus.MessageAndSessionPump - Registering message handler on entity 'demoinboxqueue' with 'MessageHandlerOptions - AutoComplete:true, MaxConcurrentCalls:5, MaxAutoRenewDuration:PT5M'
2019-11-11 15:01:55.107 INFO   ure.servicebus.MessageReceiver - Creating MessageReceiver to entity 'demoinboxqueue', ReceiveMode 'PEEKLOCK'
2019-11-11 15:01:55.117 INFO   primitives.CoreMessageReceiver - Creating core message receiver to 'demoinboxqueue'
2019-11-11 15:01:55.138 INFO   primitives.RequestResponseLink - Opened requestresponselink to $cbs
2019-11-11 15:01:55.139 INFO   us.primitives.MessagingFactory - Created CBS link to $cbs
2019-11-11 15:01:55.228 INFO   primitives.CoreMessageReceiver - Creating receive link to 'demoinboxqueue'
2019-11-11 15:01:55.251 INFO   primitives.CoreMessageReceiver - Receive link to 'demoinboxqueue' opened.
2019-11-11 15:01:55.251 INFO   ure.servicebus.MessageReceiver - Created MessageReceiver to entity 'demoinboxqueue'
2019-11-11 15:01:55.253 INFO   ure.servicebus.MessageReceiver - Created MessageBrowser to entity 'demoinboxqueue'
2019-11-11 15:01:55.254 INFO   rvicebus.MessageAndSessionPump - Created MessageReceiver to entity 'demoinboxqueue'
2019-11-11 15:02:16.474 WARN   primitives.CoreMessageReceiver - Receive link 'Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' to 'demoinboxqueue', sessionId 'null' closed with error.
com.microsoft.azure.servicebus.primitives.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The link 'G8:15665319:Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' is force detached. Code: consumer(link4236931). Details: InnerMessageReceiver was closed. TrackingId:67c04b850000001c0140a6835dc969d3_G8_B2, SystemTracker:demoExampleSB:Queue:demoinboxqueue, Timestamp:2019-11-11T14:02:16
	at com.microsoft.azure.servicebus.primitives.ExceptionUtil.toException(ExceptionUtil.java:53)
	at com.microsoft.azure.servicebus.primitives.CoreMessageReceiver.onClose(CoreMessageReceiver.java:784)
	at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:66)
	at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:42)
	at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176)
	at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
	at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291)
	at com.microsoft.azure.servicebus.primitives.MessagingFactory$RunReactor.run(MessagingFactory.java:491)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.microsoft.azure.servicebus.amqp.AmqpException: The link 'G8:15665319:Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' is force detached. Code: consumer(link42326931). Details: InnerMessageReceiver was closed. TrackingId:67c04b850000001c0140a6835dc969d3_G8_B2, SystemTracker:demoExampleSB:Queue:demoinboxqueue, Timestamp:2019-11-11T14:02:16
	... 10 common frames omitted
2019-11-11 15:02:16.474 INFO   primitives.CoreMessageReceiver - Receive link 'Receiver_47a5f0_1ae2eed451734252a5c4e6128a4e1523_G8' to 'demoinboxqueue', sessionId 'null' will be reopened after 'PT1S'
2019-11-11 15:02:17.475 INFO   primitives.CoreMessageReceiver - Recreating receive link to 'demoinboxqueue'
2019-11-11 15:02:17.488 INFO   primitives.CoreMessageReceiver - Creating receive link to 'demoinboxqueue'
2019-11-11 15:02:17.499 INFO   primitives.CoreMessageReceiver - Receive link to 'demoinboxqueue' opened.

<dependency>
   <groupId>com.microsoft.azure</groupId>
   <artifactId>azure-servicebus</artifactId>
   <version>3.0.0</version>
</dependency>

[hemanttanwar]

@marvinstorage Thank You fir the detail provided. It was helpful. I am able to replicate the issue. We will update you as we investigate this issue. This issue exists with following version also. <dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-servicebus</artifactId> <version>3.1.2</version> </dependency>

[yvgopal]

It is not a bug. I ran the test myself. The issue is with the message handler options you passed in.

new MessageHandlerOptions(5, true, Duration.ofMinutes(5L), Duration.ofMinutes(60) )

The last parameter says wait for one hour before a receive request is timed out. When a link is closed and reopened, already waiting receive requests are let to time out and new receive requests will send more credits to receive messages. (You can argue why it is doing like that, but that is a different issue). In your case, all 5 concurrent receive calls will timeout after 1 hour and then they will fire receive requests again. So after 1 hour, your application will start receiving messages again.

If you don’t pass in any argument for that parameter, it will default to 1 minute. You can try your reproduction example with different wait times and see the difference for yourself.

I am closing it. Reopen the issue if you disagree.