[Question] Large additional financial cost due to useless logs

Library name and version

Azure.Messaging.ServiceBus 7.15.0

Query/Question

Could you please explain why in the class ServiceBusRetryOptions the property TryTimeout has a check that does not allow us to specify the value TimeSpan.MaxValue? Are there any reasons that do not allow us to turn off this timer for Azure.Messaging.ServiceBus? I would be very grateful if you give an example of when disabling the timer can lead to unpleasant consequences, this example will help a lot in understanding and we can use your package more correctly.

The fact is that Azure.Messaging.ServiceBus uses Microsoft.Azure.Amqp in which there are two options for initializing the message processing pipeline: by timer and by signal. Every time the timer expires, a message processing pipeline is started for each consumer, even if there are no messages in the queue, which as a result leads to a lot of meaningless logs inside ApplicationInsights. In Microsoft.Azure.Amqp it is possible to disable the timer, for this, you just need to specify the TimeSpan.MaxValue value. By default, the timer is set to 1 minute, so one log is created every minute for each consumer. If the project has 100 consumers, then every minute there will be 100 extra logs, 6 000 in one hour, 144 000 in one day, 4 320 000 in a month… This is an insane amount that requires additional financial costs. Changing the timer to 1 hour doesn’t solve the problem, only slightly lessens it.

Since we have a very large project on SOA architecture, we have tens and hundreds of consumers, so this is a serious problem for us.

I made a small test project that you can use to reproduce the problem. You just need to run it, wait 5-10 minutes and you will see such a bunch of logs.

Environment

Windows 11 (10.0.22621) dotnet SDK 6.0.410 Microsoft.ApplicationInsights.AspNetCore 2.22.0-beta3 (reproduced in all versions)

[Shazhko-Artem]

@lmolkova Thank you very much for the answer. We cannot disable all Azure SDK requests and dependencies, but we considered the option of writing a filter. We have made the following filter.

public class ServiceBusReceiverFilter : ITelemetryProcessor
{
    private readonly ITelemetryProcessor next;

    public ServiceBusReceiverFilter(ITelemetryProcessor next)
    {
        this.next = next;
    }

    public void Process(ITelemetry telemetry)
    {
        if (telemetry is DependencyTelemetry { Name: "ServiceBusReceiver.Receive" })
        {
            return;
        }

        this.next.Process(telemetry);
    }
}

Initially, we were worried that by filtering telemetry by the name ServiceBusReceiver.Receive, we might accidentally filter out another important telemetry with the same name, but with different content.

As we found out, without a filter, this log is written within this DiagnosticScope and displays the time that has passed since the previous call to the ReceiveMessagesAsync method. (The consumer received messages every 10-11 seconds and it created telemetry records with a duration of 10 seconds.)

In general, this filter works for us, we will conduct a few more additional tests, and if everything goes well, I will close this ticket. Thank you very much for your support.

[lmolkova]

@Shazhko-Artem

you can disable all Azure SDK requests and dependencies by configuring DependencyTrackingTelemetryModule with the following code

builder.Services.ConfigureTelemetryModule<DependencyTrackingTelemetryModule>(
    (m, _) => m.EnableAzureSdkTelemetryListener = false);

You can also filter out these timed-out operations using telemetry processors.

We trace these calls since they tell that consumers are trying to receive messages and get nothing, which is still valuable information. The rate of them (2 dependencies per minute and subject to sampling would be quite low compared to the rest of the telemetry services under scale would generate).