question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PrefetchCount not work as expected

See original GitHub issue

Describe the bug

For some overloaded methods of IBus.Advanced.Consume , the prefetchcount parameter is not work as expected

To Reproduce


string MQConnectionStrFormat =
    "host={0};virtualHost={1};username={2};password={3};prefetchcount=10;publisherConfirms=true";

string conn = string.Format(MQConnectionStrFormat, "********", "****", "****","********");

var _bus = RabbitHutch.CreateBus(conn);

var queue=await _bus.Advanced.QueueDeclareAsync("testprefetch"
);
var exchange = _bus.Advanced.ExchangeDeclare("testprefetch", ExchangeType.Direct);
_=_bus.Advanced.Bind(exchange, queue, "testprefetch");


// publish test messages
Enumerable.Range(1, 20).ToList().ForEach( async i =>
{
    var msg = new Message<string>("prefetch msg " + i);
    await _bus.Advanced.PublishAsync(exchange, "testprefetch",false,msg);
});

then subscribe handlers in several ways

//work as expected , it consume 10 message per 5 seconds
_bus.Advanced.Consume(queue,  async (data, mp, mri) =>
{
     var message = Encoding.UTF8.GetString(data.ToArray());
     Console.WriteLine(message);
     await Task.Delay(5000);
     return AckStrategies.Ack;
 }, config =>
 {
     config.WithPrefetchCount(10);
 });
// not work as expected, it consume 1 message per 5 second
_bus.Advanced.Consume(queue,  async (data, mp, mri) =>
{
    var message = Encoding.UTF8.GetString(data.ToArray());
    Console.WriteLine(message);
    Task.Delay(5000).Wait();
    return AckStrategies.Ack;
}, config =>
{
    config.WithPrefetchCount(10);
});
// not work as expected, it consume 1 message per 5 second
_bus.Advanced.Consume(queue, (data, mp, mri) =>
{
    var message = Encoding.UTF8.GetString(data.ToArray());
    Console.WriteLine(message);
    Task.Delay(5000).Wait();
    return Task.FromResult(AckStrategies.Ack);
}, config =>
{
    config.WithPrefetchCount(10);
});

Only the first of the above methods can work as expected, which makes me very confused, Also, I dont know the difference between the prefetchcount parameter within the connection string and the WithPrefetchCount configured in the subscribe method, so i set both to 10…

I don’t know what caused the failure of the following two methods, which makes me very worried about unexpected problems in the program I wrote

So here again to summarize my question:

  1. What could cause the prefetchcount parameter to not work?
  2. wht is the difference between the prefetchcount parameter within the connection string and the WithPrefetchCount configured in the subscribe method?

Please complete the following information):

  • EasyNetQ version: [e.g. 7.3.7]
  • RabbitMQ version [e.g. 3.8.18]

Issue Analytics

  • State:closed
  • Created 9 months ago
  • Reactions:1
  • Comments:9 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
Plinercommented, Jan 4, 2023

for my second question , after reading this issue https://github.com/EasyNetQ/EasyNetQ/issues/1467 , I guess the prefetchcount in the connection string is used to control the concurrent limit of messages sent by RabbitMQ to EasyNetQ for processing, and WithPrefetchCount method is used by EasyNetQ to set the concurrency limit to call Handler after receiving messages from rabbitmq , is my understanding correct?

I prefer to think about PrefetchCount as a count of messages which are prefetched from a queue without any knowledge of how many “threads” are used for processing. ConcurrencyLimit should be used to set up count of threads for processing.

Having said that, at the moment ConcurrencyLimit could be set only for all models of a connection, i.e. globally in case of EasyNetQ. RabbitMQ.Client needs to be patched to allow it on a per model basis.

1reaction
Plinercommented, Jan 4, 2023

If you change code slightly, it will work as expected.

// 2nd example
_bus.Advanced.Consume(queue, async (data, mp, mri) =>
{
    await Task.Yield(); // Line changed
    var message = Encoding.UTF8.GetString(data.ToArray());
    Console.WriteLine(DateTime.UtcNow + " " + message);
    Task.Delay(5000).Wait();
    return AckStrategies.Ack;
}, config => { config.WithPrefetchCount(10); });

So the issue here is the following: if a message callback is executed synchronously(even if it is marked as async), then it could block dispatcher.

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/6.x/projects/RabbitMQ.Client/client/impl/AsyncConsumerWorkService.cs#L153

Read more comments on GitHub >

github_iconTop Results From Across the Web

PubSub.SubscribeAsync prefetchCount does not work as ...
Hello,. Maybe I just missed some breaking change but it looks like in EasyNetQ 7 prefetchCount does not work the same way it...
Read more >
django - Prefetch Related not working as expected
I'm developing an API with Django 1.11.11 and python 3.6.4. I have the following model: class Subsection(models.Model): genres = models.
Read more >
Prefetch Azure Service Bus messages
If there are no messages available for delivery, the receive operation empties the buffer and then waits or blocks, as expected. Why is...
Read more >
FAQ: How to Optimize the RabbitMQ Prefetch Count
Prefetching in RabbitMQ simply allows you to set a limit of the number of unacked (not handled) messages. There are two prefetch options ......
Read more >
Prefetch_factor not working
Hi @ptrblck, I don't understand the exact use of the prefetch factor argument. Acc. to the docs, it is Number of batches loaded...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found