Concurrency causing failure with connection limit

Regarding https://github.com/jbielick/faktory_worker_node/blob/1ae4482529fb70dc276be68d39ce10f2ea44cc31/lib/worker.js#L59

Is there a reason the poolSize for the client is set to concurrency + 2? We have concurrency set to 100 and have many queues open and this is causing Faktory to essentially grind to a halt to the point where using the Web UI does not work. We also use Elixir and the concurrency there is not a problem.

Unless each Redis connection is blocking while the job is running, this seems like overkill.

[emhagman]

Thanks @jbielick. Since the max time for FETCH when no job exists is 2 seconds and the aquire timeout is 5 seconds, this should not be too much of an issue as long as I keep it around concurrency * 0.5. Appreciate the change!

[mperham]

Sidekiq <4 used a single fetch thread which made it very sensitive to Redis network latency. Sidekiq 4+ uses $CONCURRENCY fetchers (each worker thread fetches from Redis directly) to minimize latency. Sidekiq Pro’s super_fetch uses 2 threads for $REASONS.

On Thu, Nov 7, 2019 at 11:23 AM Josh Bielick [e-mail] wrote:

I thought I was copying the one fetch per processor concept from the sidekiq source, is that not how it works? Any other best practices in worker library design?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jbielick/faktory_worker_node/issues/37?email_source=notifications&email_token=AAAAWX4O3ZZZQB3BLSLON6TQSRTKDA5CNFSM4JJXUJQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDNQUJY#issuecomment-551225895, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWX7BUM33KSAI5W52JADQSRTKDANCNFSM4JJXUJQA .