[BUG] [SB] SynchronousReceiverClient stops abruptly by logging the error 'bound must be greater than origin' and couldn't recover.

Describe the bug One of our customer’s environment had encountered issue with Service Bus Receiver using Topic-Subscription. We developed the receiver using Synchronous Receiver Client using Azure Service Bus Java SDK 7.10.1 with OAuth as follows: However after certain period of time, the client encounters errors and couldn’t recover using the retry config

Exception or Stack Trace 2022-08-31T11:58:33,509 ERROR [reactor-executor-1] c.a.c.amqp.implementation.RetryUtil - Failed to create receive link horizon.buy.out.po.common.sd.t/subscriptions/liquor_food.wms.po_73f434_1661907172730 bound must be greater than origin java.lang.IllegalArgumentException: bound must be greater than origin at java.base/java.util.concurrent.ThreadLocalRandom.nextLong(ThreadLocalRandom.java:388) at reactor.util.retry.RetryBackoffSpec.lambda$generateCompanion$4(RetryBackoffSpec.java:599) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:374) at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250) at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:97) at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:189) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) at reactor.core.publisher.Operators.error(Operators.java:197) at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) at reactor.core.publisher.Mono.subscribe(Mono.java:4150) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) at reactor.core.publisher.Operators.error(Operators.java:197) at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:71) at reactor.core.publisher.Mono.subscribe(Mono.java:4150) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:270) at reactor.core.publisher.MonoWhen$WhenCoordinator.signalError(MonoWhen.java:172) at reactor.core.publisher.MonoWhen$WhenInner.onError(MonoWhen.java:288) at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:141) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2062) at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:202) at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:270) at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) at com.azure.core.amqp.implementation.RequestResponseChannel.terminateUnconfirmedSends(RequestResponseChannel.java:479) at com.azure.core.amqp.implementation.RequestResponseChannel.onTerminalState(RequestResponseChannel.java:435) at com.azure.core.amqp.implementation.RequestResponseChannel.lambda$new$3(RequestResponseChannel.java:190) at reactor.core.publisher.LambdaSubscriber.onComplete(LambdaSubscriber.java:132) at reactor.core.publisher.FluxDistinctUntilChanged$DistinctUntilChangedSubscriber.onComplete(FluxDistinctUntilChanged.java:172) at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replayNormal(FluxReplay.java:805) at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replay(FluxReplay.java:898) at reactor.core.publisher.ReplayProcessor.tryEmitComplete(ReplayProcessor.java:465) at reactor.core.publisher.SinkManySerialized.tryEmitComplete(SinkManySerialized.java:61) at reactor.core.publisher.InternalManySink.emitComplete(InternalManySink.java:68) at com.azure.core.amqp.implementation.handler.Handler.close(Handler.java:155) at com.azure.core.amqp.implementation.handler.LinkHandler.handleRemoteLinkClosed(LinkHandler.java:121) at com.azure.core.amqp.implementation.handler.LinkHandler.onLinkRemoteClose(LinkHandler.java:61) at com.azure.core.amqp.implementation.handler.ReceiveLinkHandler.onLinkRemoteClose(ReceiveLinkHandler.java:219) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:91) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

To Reproduce Create a Topic Subscription receiver application with Synchronous receiver client. and keep sending messages to Topics using another application at regular intervals of time.

Code Snippet

AmqpRetryOptions retry = new AmqpRetryOptions();
Duration delay = Duration.ofMillis(1000);
Duration timeout = Duration.ofMillis(60000);
retry.setMaxRetries(10);
retry.setDelay(delay);
retry.setMode(AmqpRetryMode.FIXED);
retry.setTryTimeout(timeout);
creds = new ClientSecretCredentialBuilder()
.tenantId(TenantID)
.clientId(ClientID).clientSecret(ClientSecret)
.build();
ServiceBusReceiverClient receiverClient= new ServiceBusClientBuilder().credential(host,creds)
                                        .connectionString(connectionString)
                                        .retryOptions(retry)
                                        .receiver().disableAutoComplete()
.topicName(queueName)
.subscriptionName(subscriberName)
.buildClient();

while(flag){
receiverClient.receiveMessages(maxMessages);
}

Expected behavior The receiver should always be up and running and if any intermittent errors occur, it should have ensured fault tolerance and recover from the error using the retry configuration

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Windows
• IDE: eclipse
• Library/Libraries: [azure-messaging-servicebus]» 7.10.1 and its dependencies
• Java version: 8
• App Server/Environment: Redhat Openshift
• Frameworks: Spring boot

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn’t provide solution for the problem, please provide:

• verbose dependency tree (mvn dependency:tree -Dverbose)
• exception message, full stack trace, and any available logs

Additional context We have had similar issue last year and worked with @conniey @anuchandy @liukun-msft . However this issue is observed in one of our customer’s environment. Besides, we haven’t not configured any throttling mechanism in this scenario.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [*] Bug Description Added
• [*] Repro Steps Added
• [*] Setup information Added xdockpurchaseorderservice-1-m2rsb-xdockpurchaseorderservice.log

[anuchandy]

I see, right, min-delay of 1 min with default max-delay will trigger the validation error.

I didn’t locate any other such corner cases; yes, please update the configuration. I’m updating the ticket title, so it’s discoverable in the future.

[msftbot[bot]]

Hi, we’re sending this friendly reminder because we haven’t heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don’t hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!