[BUG] EventHubBufferedProducerClient: high CPU usage when no events are being sent

Library name and version

Azure.Messaging.EventHubs 5.7.0

Describe the bug

We have an application with high event producing rate (at least hundreads per second) and after a load test (peak of 25K events in 1 minute) with duration of 3 minutes, we observed that CPU usage keeps almost the same as the event ‘processing’.

We compared the very same approach/code/test exchanging to an EventHubProducerClient (on same version) and CPU drops seconds after the ‘injection’ of events finishes. In both versions, each queue name has a unique instance of this class/Producer being (re)used to send all events.

Expected behavior

After the load test has stopped - as the the production of events has stopped too - we expect the CPU usage drops to same levels before the test, after a few seconds.

Actual behavior

CPU usage keeps at the almost same level after several minutes of the ending of load test. Only after a process restart the CPU usage has got back to normal levels.

Reproduction Steps

The code is very straighforward (I have edited to remove some internal abstraction details, but nothing relevant):

• Using EventHubBufferedProducerClient:

    public class SenderQueue<T> : IAsyncDisposable
    {
        private readonly ISerializer<T> _serializer;
        private readonly EventHubBufferedProducerClient _eventHubProducerClient;
        private readonly ILogger _logger;

        public SenderQueue(
            IConfiguration configuration,
            string queueName,
            ISerializer<T> serializer,
            ILogger logger)
        {
            _serializer = serializer;
            _logger = logger;
            _eventHubProducerClient = GetProducerClient(configuration, queueName);
        }

        public virtual async Task EnqueueAsync(T item, CancellationToken cancellationToken = default)
        {
            if (item == null)
                throw new ArgumentNullException(nameof(item));
           var serializedItem = _serializer.Serialize(item);
           await _eventHubProducerClient.EnqueueEventAsync(new EventData(serializedItem), cancellationToken);
        }

        public async ValueTask DisposeAsync()
        {
            _eventHubProducerClient.SendEventBatchSucceededAsync -= SendSuccessfulHandler;
            _eventHubProducerClient.SendEventBatchFailedAsync -= SendFailedHandler;
            await _eventHubProducerClient.DisposeAsync();
        }

        protected EventHubBufferedProducerClient GetProducerClient(IConfiguration configuration, string queueName)
        {
            var eventHubBufferedProducerClient = new EventHubBufferedProducerClient(
                            configuration.AzureEventHubConnectionString,
                            queueName);
            eventHubBufferedProducerClient.SendEventBatchSucceededAsync += SendSuccessfulHandler;
            eventHubBufferedProducerClient.SendEventBatchFailedAsync += SendFailedHandler;

            return eventHubBufferedProducerClient;
        }

        private Task SendSuccessfulHandler(SendEventBatchSucceededEventArgs args)
        {
            _logger.Verbose("Successfully sent {EventBatchCount} events of {ItemType} to EventHub", args.EventBatch.Count, typeof(T).Name);
            return Task.CompletedTask;
        }

        private Task SendFailedHandler(SendEventBatchFailedEventArgs args) 
        {
            _logger.Warning(args.Exception, "Failed to send {EventBatchCount} events of {ItemType} to EventHub", args.EventBatch.Count, typeof(T).Name);
            return Task.CompletedTask;
        }
    }

• Using EventHubProducerClient:

    public class SenderQueue<T> : IAsyncDisposable
    {
        private readonly ISerializer<T> _serializer;
        private readonly EventHubProducerClient _eventHubProducerClient;
        private readonly ILogger _logger;

        public SenderQueue(
            IConfiguration configuration,
            string queueName,
            ISerializer<T> serializer,
            ILogger logger)
        {
            _serializer = serializer;
            _logger = logger;
            _eventHubProducerClient = GetProducerClient(configuration, queueName);
        }

        public virtual async Task EnqueueAsync(T item, CancellationToken cancellationToken = default)
        {
            if (item == null)
                throw new ArgumentNullException(nameof(item));
            var eventArray = BuildEventData(item);
            await _eventHubProducerClient.SendAsync(eventArray, cancellationToken);
        }

        public async ValueTask DisposeAsync() => await _eventHubProducerClient.DisposeAsync();

        protected EventHubProducerClient GetProducerClient(IConfiguration configuration, string queueName)
        {
            return new EventHubProducerClient(
                configuration.AzureEventHubConnectionString,
                queueName);
        }

        private EventData[] BuildEventData(T item)
        {
            var serializedItem = _serializer.Serialize(item);
            return new[] { new EventData(serializedItem) };
        }
    }

Environment

• Windows Server 2016 (OS Version: 10.0.14393)
• .Net Core 3.1 (runtime Microsoft.NETCore.App 3.1.1)
• Visual Studio 2022

[jsquire]

Confirming that we’ve traced this back to a bug in which the task responsible for inspecting the buffers is not backing off during idle periods due to an error in the condition. A fix is in progress.

[andreminelli]

Awesome, thank you!