Best way to handle Schedulers in EventHubProducerAsyncClient and EventHubConsumerAsyncClient

Query/Question Hello Azure team,

First of all, please excuse me if this may seem a newbie question but it will take me some time to adopt the logic of my brain from synchronous world to asynchronous world 😃 .

I am now writing Camel EventHub component and this time, I’d like to use the async clients without any blocking needed. However, I’d like to add custom Schedulers to handle my non-blocking requests. Now taking this code example:

producerAsyncClient.send(Arrays.asList(new EventData("first 1"), new EventData("first 2")))
                .publishOn(Schedulers.newSingle("publish"))
                .subscribeOn(Schedulers.newSingle("subscribe"))
                .subscribe(
                        body -> {System.out.println("Thread: " + Thread.currentThread());},
                        error -> System.out.println("Error " + error.getMessage() + " on thread: " + Thread.currentThread().getName()),
                        () -> {
                            System.out.println("done all");
                            System.out.println("Thread: " + Thread.currentThread().getName());
                        });

Which will print the following logs:

2020-08-06 16:06:41,376 [main           ] INFO  EventHubConnectionProcessor    - namespace[testcamel.servicebus.windows.net] entityPath[testhub]: Next AMQP channel received, updating 0 current subscribers
2020-08-06 16:06:41,395 [subscribe-1    ] INFO  ReactorConnection              - connectionId[MF_8b659d_1596722801357]: Creating and starting connection to testcamel.servicebus.windows.net:5671
2020-08-06 16:06:41,418 [subscribe-1    ] INFO  ReactorExecutor                - connectionId[MF_8b659d_1596722801357], message[Starting reactor.]
2020-08-06 16:06:41,428 [single-2       ] INFO  ConnectionHandler              - onConnectionInit hostname[testcamel.servicebus.windows.net], connectionId[MF_8b659d_1596722801357]
2020-08-06 16:06:41,428 [single-2       ] INFO  ReactorHandler                 - connectionId[MF_8b659d_1596722801357] reactor.onReactorInit
2020-08-06 16:06:41,428 [single-2       ] INFO  ConnectionHandler              - onConnectionLocalOpen hostname[testcamel.servicebus.windows.net:5671], connectionId[MF_8b659d_1596722801357], errorCondition[null], errorDescription[null]
2020-08-06 16:06:41,428 [subscribe-1    ] DEBUG ReactorSession                 - Connection state: UNINITIALIZED
2020-08-06 16:06:41,429 [subscribe-1    ] DEBUG EventHubReactorAmqpConnection  - Get or create producer for path: 'testhub'
2020-08-06 16:06:41,430 [subscribe-1    ] INFO  AzureTokenManagerProvider      - Creating new token manager for audience[amqp://testcamel.servicebus.windows.net/testhub], resource[testhub]
2020-08-06 16:06:41,486 [single-2       ] DEBUG SessionHandler                 - onSessionLocalOpen connectionId[MF_8b659d_1596722801357], entityName[testhub], condition[Error{condition=null, description='null', info=null}]
2020-08-06 16:06:41,489 [single-2       ] INFO  ConnectionHandler              - onConnectionBound hostname[testcamel.servicebus.windows.net], connectionId[MF_8b659d_1596722801357]
2020-08-06 16:06:41,737 [single-2       ] INFO  ConnectionHandler              - onConnectionRemoteOpen hostname[testcamel.servicebus.windows.net:5671], connectionId[MF_8b659d_1596722801357], remoteContainer[1f9c0abd70904ba4b3501556bd18fa64_G4]
2020-08-06 16:06:41,737 [single-2       ] DEBUG ReactorConnection              - connectionId[MF_8b659d_1596722801357]: Connection state: ACTIVE
2020-08-06 16:06:41,737 [single-2       ] INFO  EventHubConnectionProcessor    - namespace[testcamel.servicebus.windows.net] entityPath[testhub]: Channel is now active.
2020-08-06 16:06:41,737 [single-2       ] INFO  SessionHandler                 - onSessionRemoteOpen connectionId[MF_8b659d_1596722801357], entityName[testhub], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
2020-08-06 16:06:41,737 [single-2       ] DEBUG ReactorSession                 - Connection state: ACTIVE
2020-08-06 16:06:41,739 [single-2       ] INFO  ReactorConnection              - Setting CBS channel.
2020-08-06 16:06:41,745 [single-2       ] DEBUG ReactorSession                 - Connection state: UNINITIALIZED
2020-08-06 16:06:41,752 [single-2       ] INFO  ReactorConnection              - Emitting new response channel. connectionId: MF_8b659d_1596722801357. entityPath: $cbs. linkName: cbs.
2020-08-06 16:06:41,752 [single-2       ] INFO  stResponseChannel<cbs-session> - namespace[MF_8b659d_1596722801357] entityPath[$cbs]: Setting next AMQP channel.
2020-08-06 16:06:41,752 [single-2       ] INFO  stResponseChannel<cbs-session> - namespace[MF_8b659d_1596722801357] entityPath[$cbs]: Next AMQP channel received, updating 0 current subscribers
2020-08-06 16:06:41,763 [single-2       ] DEBUG SessionHandler                 - onSessionLocalOpen connectionId[MF_8b659d_1596722801357], entityName[cbs-session], condition[Error{condition=null, description='null', info=null}]
2020-08-06 16:06:41,764 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,765 [single-2       ] DEBUG SendLinkHandler                - onLinkLocalOpen connectionId[MF_8b659d_1596722801357], linkName[cbs:sender], localTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
2020-08-06 16:06:41,766 [single-2       ] INFO  ReceiveLinkHandler             - onLinkLocalOpen connectionId[MF_8b659d_1596722801357], linkName[cbs:receiver], localSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
2020-08-06 16:06:41,785 [single-2       ] INFO  SessionHandler                 - onSessionRemoteOpen connectionId[MF_8b659d_1596722801357], entityName[cbs-session], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
2020-08-06 16:06:41,785 [single-2       ] DEBUG ReactorSession                 - Connection state: ACTIVE
2020-08-06 16:06:41,785 [single-2       ] INFO  SendLinkHandler                - onLinkRemoteOpen connectionId[MF_8b659d_1596722801357], linkName[cbs:sender], remoteTarget[Target{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
2020-08-06 16:06:41,785 [single-2       ] INFO  stResponseChannel<cbs-session> - namespace[MF_8b659d_1596722801357] entityPath[$cbs]: Channel is now active.
2020-08-06 16:06:41,785 [single-2       ] DEBUG SendLinkHandler                - onLinkFlow connectionId[MF_8b659d_1596722801357], linkName[cbs:sender], unsettled[0], credit[100]
2020-08-06 16:06:41,785 [single-2       ] INFO  ReceiveLinkHandler             - onLinkRemoteOpen connectionId[MF_8b659d_1596722801357], linkName[cbs:receiver], remoteSource[Source{address='$cbs', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
2020-08-06 16:06:41,785 [single-2       ] INFO  stResponseChannel<cbs-session> - namespace[MF_8b659d_1596722801357] entityPath[$cbs]: Channel is now active.
2020-08-06 16:06:41,786 [single-2       ] DEBUG RequestResponseChannel<cbs>    - Scheduling on dispatcher. Message Id 1
2020-08-06 16:06:41,786 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,789 [single-2       ] DEBUG SendLinkHandler                - onLinkFlow connectionId[MF_8b659d_1596722801357], linkName[cbs:sender], unsettled[0], credit[99]
2020-08-06 16:06:41,812 [single-2       ] DEBUG RequestResponseChannel<cbs>    - Settling message: 1
2020-08-06 16:06:41,815 [single-2       ] INFO  ActiveClientTokenManager       - Scheduling refresh token task. scopes[amqp://testcamel.servicebus.windows.net/testhub]
2020-08-06 16:06:41,821 [single-2       ] DEBUG ReceiveLinkHandler             - onDelivery connectionId[MF_8b659d_1596722801357], linkName[cbs:receiver], updatedLinkCredit[0], remoteCredit[0], remoteCondition[Error{condition=null, description='null', info=null}], delivery.isPartial[false]
2020-08-06 16:06:41,822 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,822 [single-2       ] INFO  ReactorSession                 - Creating a new sender link with linkName testhub
2020-08-06 16:06:41,826 [single-2       ] DEBUG ReactorSender                  - [testhub] Connection state: UNINITIALIZED
2020-08-06 16:06:41,827 [single-2       ] DEBUG ReactorSender                  - Token refreshed: ACCEPTED
2020-08-06 16:06:41,828 [single-2       ] DEBUG SendLinkHandler                - onLinkLocalOpen connectionId[MF_8b659d_1596722801357], linkName[testhub], localTarget[Target{address='testhub', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
2020-08-06 16:06:41,848 [single-2       ] INFO  SendLinkHandler                - onLinkRemoteOpen connectionId[MF_8b659d_1596722801357], linkName[testhub], remoteTarget[Target{address='testhub', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
2020-08-06 16:06:41,848 [single-2       ] DEBUG ReactorSender                  - [testhub] Connection state: ACTIVE
2020-08-06 16:06:41,852 [single-2       ] DEBUG EventHubProducerAsyncClient    - Sending batch with size[2] to be distributed round-robin in service.
2020-08-06 16:06:41,855 [single-2       ] DEBUG EventHubReactorAmqpConnection  - Get or create producer for path: 'testhub'
2020-08-06 16:06:41,855 [single-2       ] DEBUG ReactorSession                 - linkName[testhub]: Returning existing send link.
2020-08-06 16:06:41,860 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,860 [single-2       ] DEBUG ReactorSender                  - Credits on link: 300
2020-08-06 16:06:41,860 [single-2       ] DEBUG SendLinkHandler                - onLinkFlow connectionId[MF_8b659d_1596722801357], linkName[testhub], unsettled[0], credit[300]
2020-08-06 16:06:41,860 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,861 [single-2       ] DEBUG ReactorSender                  - entityPath[testhub], linkName[testhub], deliveryTag[7d8f336858f14565be5b116433d981a8]: Sent message
2020-08-06 16:06:41,861 [single-2       ] DEBUG ReactorSender                  - Credits on link: 299
2020-08-06 16:06:41,861 [single-2       ] DEBUG SendLinkHandler                - onLinkFlow connectionId[MF_8b659d_1596722801357], linkName[testhub], unsettled[1], credit[299]
2020-08-06 16:06:41,861 [single-2       ] DEBUG DispatchHandler                - Running task for event: %s
2020-08-06 16:06:41,897 [single-2       ] DEBUG SendLinkHandler                - onDelivery connectionId[MF_8b659d_1596722801357], linkName[testhub], unsettled[1], credit[299], deliveryState[Accepted{}], delivery.isBuffered[false], delivery.id[7d8f336858f14565be5b116433d981a8]
2020-08-06 16:06:41,897 [single-2       ] DEBUG ReactorSender                  - entityPath[testhub], linkName[testhub], deliveryTag[7d8f336858f14565be5b116433d981a8]: process delivered message
done all
Thread: publish-3
2020-08-06 16:06:46,393 [main           ] INFO  EventHubConnectionProcessor    - Upstream connection publisher was completed. Terminating processor.
2020-08-06 16:06:46,393 [main           ] INFO  EventHubConnectionProcessor    - namespace[testcamel.servicebus.windows.net] entityPath[testhub]: AMQP channel processor completed. Notifying 0 subscribers.
2020-08-06 16:06:46,394 [main           ] INFO  EventHubReactorAmqpConnection  - connectionId[MF_8b659d_1596722801357]: Disposing of connection.
2020-08-06 16:06:46,394 [main           ] INFO  ReactorConnection              - connectionId[MF_8b659d_1596722801357]: Disposing of ReactorConnection.
2020-08-06 16:06:46,395 [main           ] INFO  EventHubConnectionProcessor    - namespace[testcamel.servicebus.windows.net] entityPath[testhub]: Channel is disposed.
2020-08-06 16:06:46,395 [main           ] INFO  ReactorConnection              - connectionId[MF_8b659d_1596722801357]: Removing session 'testhub'
2020-08-06 16:06:46,395 [main           ] INFO  ReactorSession                 - sessionId[testhub]: Disposing of session.
2020-08-06 16:06:46,396 [main           ] INFO  ReactorConnection              - connectionId[MF_8b659d_1596722801357]: Removing session 'cbs-session'
2020-08-06 16:06:46,396 [main           ] INFO  ReactorSession                 - sessionId[cbs-session]: Disposing of session.

While I get why the done subscription method which is printing done all uses the publish thread from publishOn, I don’t get why for example ReactorConnection initiated from thread subscribe-1 that was set from subscribeOn(Schedulers.newSingle("subscribe")), although I was reading a lot of info about publishOn vs subscribeOn online, I fail to understand how can use them for my benefit in the context of AzureAsync clients, would it possible if you can clarify how would be the optimal way to handle custom Schedulers here?

Also, from the logs, I see EventHubProducerAsyncClient is sending the events from single-2 thread, is this because of the AMQP connection initiated on a single thread Scheduler?

Thank you, Omar

[joshfree]

Thanks for filing the question, @omarsmak. @conniey from the event hubs sdk team will follow up shortly.

[omarsmak]

nevermind, I think I figured out.

Thank you