[BUG] EventHubProducerClient is being throttled but not informing the calling application.

Describe the bug EventHubProducerClient send requests throttled but without giving the information back to the client. Instead only a WARNING log is printed (where how much time to wait is also printed). Application cannot reduce the frequency of send unless the information is available to the application.

Also, the error code: 50002 (mentioned in the WARNING log) doesn’t have an entry in the linked website: https://aka.ms/sbResourceMgrExceptions (also mentioned in the WARNING log).

Exception or Stack Trace

06:50:05 [single-1] WARN  c.a.c.a.i.ReactorSender - entityPath[dummy], linkName[dummy], deliveryTag[350acd586c70467ca4f81c30714a5403]: Delivery rejected. [Rejected{error=Error{condition=com.microsoft:server-busy, description='The request was terminated because the entity is being throttled. Error code : 50002. Please wait 4 seconds and try again. To know more visit https://aka.ms/sbResourceMgrExceptionsS:N:<NAMESPACE>:EVENTHUB:DUMMY~32766,CL:3159,CC:3964,ACC:471701,LUR:<NAMESPACE>_105187,RC:991 TrackingId:0fc2c77200009ae30008e7b55eb65120_G25_B39, SystemTracker:<namespace>:eventhub:dummy~32766, Timestamp:2020-05-09T06:50:05', info=null}}]

To Reproduce Create an EventHub namespace with 1 EventHub having only 1 partition. Start consuming from an EventHub which has a lot of messages available using the same SDK’s EventProcessorClient with the following EventHandler implementation:

import com.azure.messaging.eventhubs.EventData;
import com.azure.messaging.eventhubs.EventDataBatch;
import com.azure.messaging.eventhubs.EventHubProducerClient;
import com.azure.messaging.eventhubs.models.CreateBatchOptions;
import com.azure.messaging.eventhubs.models.EventContext;
import lombok.extern.slf4j.Slf4j;

import java.util.ArrayList;
import java.util.List;
import java.util.function.Consumer;


@Slf4j
public class Event implements Consumer<EventContext>, AutoCloseable {

    private final EventHubProducerClient eventHubProducerClient;
    private final CreateBatchOptions createBatchOptions;

    private final ThreadLocal<List<EventData>> toBePushed = ThreadLocal.withInitial(ArrayList::new);
    private final ThreadLocal<Integer> approxBatchSizeInBytes = ThreadLocal.withInitial(() -> 0);

    private static final int overheadPerBatchInBytes = 15360;
    private static final int overheadPerMsgInBytes = 30;
    private static final int maxMsgsPerBatch = 990;


    public Event(EventHubProducerClient eventHubProducerClient,
                 CreateBatchOptions createBatchOptions) {

        this.eventHubProducerClient = eventHubProducerClient;
        this.createBatchOptions = createBatchOptions;
    }


    private synchronized void pushToEH() {

        try {
            EventDataBatch eventDataBatch = eventHubProducerClient.createBatch(createBatchOptions);

            for (EventData eventData: toBePushed.get()) {
                if (!eventDataBatch.tryAdd(eventData)) {

                    eventHubProducerClient.send(eventDataBatch);
                    eventDataBatch = eventHubProducerClient.createBatch(createBatchOptions);

                    if (!eventDataBatch.tryAdd(eventData)) {
                        log.error("Event is too large for an empty batch. Max size: " + eventDataBatch.getMaxSizeInBytes());
                    }
                }
            }
            toBePushed.get().clear();

            if (eventDataBatch.getCount() > 0) {
                eventHubProducerClient.send(eventDataBatch);
            }
        } catch (Exception e) {
            log.error("Exception: ", e);
        }

        approxBatchSizeInBytes.set(0);
    }


    private void publish(byte[] payload) {

        toBePushed.get().add(new EventData(payload));
        approxBatchSizeInBytes.set(approxBatchSizeInBytes.get() + (payload.length + overheadPerMsgInBytes));

        if (toBePushed.get().size() > maxMsgsPerBatch
                || (approxBatchSizeInBytes.get() + overheadPerBatchInBytes >= createBatchOptions.getMaximumSizeInBytes())) {

            pushToEH();
        }
    }


    @Override
    public void accept(EventContext eventContext) {

        try {
            publish(eventContext.getEventData().getBody());
        } catch (Exception e) {
            log.error("Error in Event Handler: ", e);
        }
    }


    @Override
    public void close() {
        eventHubProducerClient.close();
        log.info("EventHubProducerClient stopped.");
    }
}

Expected behavior The throttling information must be passed back to the calling application so the application can throttle the behavior at its own end.

Setup (please complete the following information):

• Version of the Library used: 5.1.0

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[srnagar]

The exception type that will be propagated in this case is AmqpException. This will have the errorCondition set to AmqpErrorCondition.SERVER_BUSY_ERROR. The wait time is currently not exposed as a field in the exception but is available in the error message - “… Please wait 4 seconds and try again”. I will see if this can be exposed as a Duration in the exception.

The error codes are language agnostic but I agree that they should be documented in a language-neutral page. Thanks for your feedback and I’ll follow up on making the documentation on error code better.

[shubhambhattar]

Thanks for the info @srnagar .